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Purpose 

The Committee will hold a hearing on NASA policy regarding the agency’s man- 
agement of the National Aviation Operations Monitoring Service (NAOMS). NAOMS 
has been in the press due to NASA’s refusal to release the data to an Associated 
Press (AP) reporter, offering the rationale that release of the information might un- 
dermine the fl 3 dng public’s confidence in the aviation system because it relates to 
safety. NASA’s refusal to release this data has been widely condemned in the Na- 
tion’s press with editorials in many papers. NASA’s Administrator Michael Griffin 
has formally distanced himself from that rationale, but he has not yet made it clear 
when or even whether NASA will publicly release this data. 

Witnesses 

Panel 1 

Dr. Michael Griffin, Administrator, National Aeronautics and Space Administra- 
tion (NASA) 

Mr. Jim Hall, Managing Partner, Hall and Associates, LLC, and Former Chairman, 
National Transportation Safety Board (NTSB) 

Panel 2 

Dr. Robert S. Dodd, Safety Consultant and President, Dodd & Associates, LLC 

Dr. Jon A. Krosnick, Frederic O. Glover Professor in Humanities and Social 
Sciences, Stanford University 

Captain Terry McVenes, Executive Air Safety Chairman, Air Line Pilots Associa- 
tion 

Background 

On October 29, Administrator Griffin sent a letter to the Committee indicating 
that the data was being provided to the Committee, but noting that “NASA believes 
that the data contains both confidential commercial data and information that could 
compromise anon 3 Tnity that should be redacted prior to public release.” Staff have 
been unable to find a NASA or Battelle staffer [the contractor on the project] who 
can articulate what commercially sensitive information resides in these data bases. 
As to anonymity, Battelle indicated that all personal identif 3 dng information was 
stripped away from the data within 24 hours of conducting a survey. It is unclear 
what data should be removed prior to public release and this may be a question for 
NASA. 

The concern NASA has expressed in its initial FOIA rejection letter was that pub- 
lic release of the data may undermine confidence in fl 3 dng among the public. How- 
ever, other data safety systems are already open to the public and include plenty 
of details that could have far more impact on public confidence than data contained 
in a spreadsheet. The best known is the Aviation Safety Reporting System (ASRS) 
which includes numerous stories about near misses in the air and on the ground. 
The bottom line is that when planes have actually crashed, people keep going right 
to the airport. The Committee asked NASA to provide all records of the aviation 
industry expressing concerns that their commercial interests could be damaged or 
objecting to the impact on the flying public’s attitudes if NAOMS data were made 
publicly available, and NASA could find no responsive records. 

In addition to the FOIA issue, the hearing will provide an opportunity for the 
Committee to learn about aviation safety data sources and the rationale behind 
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launching NAOMS in the first place. All other data systems involve voluntary self- 
reporting tied to either incidents that have happened or else data that has been fil- 
tered by private parties to strip information out of the report prior to being turned 
over to the government. FAA collects most of these data sources; NASA manages 
the Aviation Safety Reporting System (ASRS) for FAA. If it had been rolled out 
operationally, NAOMS would have integrated continuous survey data from pilots, 
ground controllers, ground crews, and cabin crews to create a complete picture of 
what is happening in the air safety system nationally. This information would not 
be driven by adverse events and would have a statistical rigor that the self-report- 
ing anecdotal systems lack. As a result, safety experts could mine the NAOMS data 
for insights into new safety threats as they emerge. 

The aviation system is changing due to new information and communications 
technologies that are being introduced into the system. It is also anticipated that 
the national airspace system will have to handle up to three times as much demand 
by 2025 compared to 2000. The voluntary reporting systems of the past may not be 
good enough, and certainly do not represent what could be achieved with improved 
data systems, to keep the skies over the United States safe. NAOMS was to be that 
pro-active, forward looking tool to identify problems tied to increasing demands on 
capacity and unexpected problems with the introduction of new technologies. 

NASA spent three years developing and field testing the NAOMS survey with 
support by Battelle and several distinguished subcontractors who were experts in 
survey methodology or aviation safety. Then NASA ran a survey of commercial pi- 
lots for almost four years. Over 24,000 pilots responded to the survey. Another 4000 
general aviation pilots were surveyed during a span of several months over 2002- 
2003. The contractor also began work to roll out a survey of air traffic controllers, 
but it was never implemented in the field. After spending more than $8 million to 
develop this tool and begin to put it in place, NASA shut it down before it became 
operational. The project enjoyed unusual success in gathering responses from pilots, 
but the project also ran up against competing priorities within the agency, as well 
as a lack of interest at the FAA. 

In shutting the project down, NASA has done absolutely nothing to either adver- 
tise the methodology and the goal they hoped to achieve or release any analytical 
products that give insights into air safety trends. This was true until the AP re- 
porter pushed to get the materials out. Only then did the top managers for this 
project at NASA begin to try to put some sort of report together. NASA says a tech- 
nical report will be released by the end of the year, but prior to a week ago, the 
report was described by both NASA counsel and NASA researchers to Committee 
staff as something that would represent analytical insights drawn from the data 
with recommendations for improving air safety. It appears that NASA has moved 
the goal posts even on this belated work product. 

The reasons that NAOMS was needed have not changed. The national air trans- 
portation system appears safe at the moment, but new technologies and stresses 
will produce exactly the situation that NAOMS was designed to help address. 

To help the Committee sort through some of this, we will receive testimony from 
Dr. Michael Griffin, the NASA Administrator. The Committee will also take testi- 
mony from Mr. Jim Hall (former head of the National Transportation Safety Board 
and member of the 1997 Aviation Safety and Security Commission — the Gore Com- 
mission), Dr. Robert Dodd (aviation safety expert who managed the NAOMS 
project under contract to Battelle), Dr. Jon Krosnik (Stanford statistics professor 
who helped design the survey), and a representative of the Airline Pilots Association 
(ALFA), Captain Terry MeVenes. ALPA actually opposes release of the raw data, 
but they do favor analysis of that information. NASA has also “handed-off’ the 
NAOMS methodology to ALPA (though it has been redesigned as a web-based, not 
phone-based survey) so that they can administer the survey to their members. How- 
ever, ALPA has told Committee staff that they have not decided what questions 
they would ask, who they would ask them of, or even when to run a survey. They 
have done nothing with NAOMS to date. 
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Chairman GORDON. I want to welcome all of you, and I especially 
want to welcome our witnesses to today’s hearing. You have made 
yourself available to testify on relatively short notice, and I appre- 
ciate your willingness to assist the Committee in carrying out our 
oversight responsibilities on this important issue. 

It was important that we met as soon as possible to get to the 
bottom of what has been going on and what NASA intends to do 
from this point forward. America’s air transportation system is crit- 
ical both to our nation’s economic vitality and to our quality of life. 

However, it is no secret that the system faces increasing stresses 
as air traffic demand continues to grow, demand that is expected 
to increase by a factor of two or three by the year 2025. And those 
stresses make it even more important that all necessary steps are 
taken to maintain air safety. It is the right thing to do, and the 
American public expects it. 

Our citizens want to be sure that the government and the avia- 
tion industry are doing all that can be done to keep the air trans- 
portation system safe. That is why both the public and Members 
of Congress alike have such a strong reaction to reports that NASA 
has been withholding an aviation safety survey database compiled 
by taxpayer dollars. NASA’s explanation for its refusal to release 
the data was both troubling and unconvincing. 

Specifically, NASA has stated the data can’t be released because, 
and I quote, “It could materially affect the public confidence in, and 
the commercial welfare of air carriers.” 

Well, as I have said before, NASA needs to focus on maintaining 
and increasing the safety of the flying public, not protecting the 
commercial air carriers. And if NASA accomplishes that and if we 
have a safe traveling environment, then the commercial air car- 
riers, their situation will certainly be enhanced. Dr. Griffin has in- 
dicated that he agrees, and he will testify today that NASA will 
publicly release the NAOMS data. 

While we need to clarify just exactly what will be released and 
when, and I hope it will be soon, I am pleased that he is taking 
that action, as his usual candor dictates. If scheduling this hearing 
helped bring about the change of direction at NASA, I think that 
it has been a constructive exercise by our oversight responsibilities. 

However, the issue we have to consider today goes beyond simply 
the release of the data NASA is withholding. We also have a ques- 
tion of priorities. As former NTSB Chairman Jim Hall will testify, 
and again, I quote, “A true safety culture requires transparency 
and consistent vigilance.” 

Numerous individuals familiar with this report have told us that 
it has envisioned, was envisioned as a long-term, continuing data 
collection and analysis effort to identify aviation accident precur- 
sors and safety trends. And several of our witnesses today will tes- 
tify that it has potential to provide information and insights 
unobtainable from existing data sources. 

Therefore, by most accounts, the report appeared to be a prom- 
ising avenue for ensuring that our nation’s air transportation sys- 
tem would retain its impressive safety record in the coming years. 
Yet whether it was due to shifting priorities, budgetary constraints, 
cultural differences between agencies, or something else, the report 
has largely been cast adrift by NASA and the FAA. 
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I hope that one outcome of today’s hearing will be the reconstruc- 
tion of the report and project by NASA and the FAA. However, I 
think we in Congress also need to take a close look at NASA’s over- 
all aviation safety program to make sure that it still addresses the 
most relevant safety questions facing the Nation’s air transpor- 
tation system. 

That is going to be one of the focuses of today’s hearings and in 
the coming months. Maintaining and improving aviation safety is 
an important task for the Federal Government to accomplish, 
working in partnership with the aviation industry. The stakes are 
high, and we need to get it right. 

We have a lot to do and to cover today, so I again welcome our 
witness at today’s hearing, and I now yield to my good friend and 
colleague. Ranking Member Ralph Hall. 

[The prepared statement of Chairman Gordon follows:] 

Prepared Statement of Chairman Bart Gordon 

Good afternoon. I’d like to welcome all of our witnesses to today’s hearing. You 
have made yourselves available to testify on relatively short notice, and I appreciate 
your willingness to assist the Committee in carrying out our oversight on this im- 
portant issue. 

It was important that we meet as soon as possible to get to the bottom of what 
has been going on, and what NASA intends to do from this point forward. America’s 
air transportation system is critical both to our nation’s economic vitality and to our 
quality of life. 

However, it’s no secret that the system faces increasing stresses as air travel de- 
mand continues to grow — demand that is expected to increase by a factor of two to 
three by 2025. And those stresses make it even more important that all necessary 
steps are taken to maintain air safety. It’s the right thing to do, and the American 
public expects it. 

Our citizens want to be sure that the government and the aviation industry are 
doing all that can be done to keep the air transportation system safe. That’s why 
both the public and Members of Congress alike had such a strong reaction to reports 
that NASA has been withholding an aviation safety survey data base compiled with 
taxpayer dollars. NASA’s explanation for its refusal to release the data was both 
troubling and unconvincing. 

Specifically, NASA was saying the data can’t be released because it “could materi- 
ally affect the public confidence in, and the commercial welfare of the air car- 
riers. . 

Well, as I’ve said before, NASA needs to focus on maintaining and increasing the 
safety of the flying public, not on protecting the commercial air carriers. Dr. Griffin 
has indicated that he agrees, and he will testify today that NASA will publicly re- 
lease the NAOMS data. 

While we need to clarify just exactly what will be released and when — and I hope 
it will be soon — I am pleased that he is taking that action. If scheduling this hear- 
ing helped bring about this change of direction at NASA, I think that it has been 
a constructive exercise of our oversight responsibilities. 

However, the issues we have to consider today go beyond simply the release of 
the data NASA is withholding. We also have a question of priorities. As former 
NTSB Chairman Jim Hall will testify today: “A true safety culture requires trans- 
parency and constant vigilance.” 

Numerous individuals familiar with the NAOMS project have told us that it was 
envisioned as a long-term, continuing data collection and analysis effort to identify 
aviation accident precursors and safety trends. And several of our witnesses today 
will testify that it has the potential to provide information and insights 
unobtainable from existing data sources. 

Thus, by most accounts, NAOMS appeared to be a promising avenue for ensuring 
that our nation’s air transportation system would retain its impressive safety record 
in the coming years. Yet whether it was due to shifting priorities, budgetary con- 
straints, cultural differences between agencies, or something else — NAOMS has 
largely been cast adrift by NASA and the FAA. 

I hope that one outcome of today’s hearing will be a reconsideration of the 
NAOMS project by NASA and the FAA. However, I think we in Congress also need 
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to take a close look at NASA’s overall aviation safety program to make sure that 
it is still addressing the most relevant safety questions facing the Nation’s air trans- 
portation system. 

That is going to be one of the focuses of this committee’s oversight in the coming 
months. 

Maintaining and improving aviation safety is an important task for the Federal 
Government to accomplish — working in partnership with the aviation industry. 

The stakes are high, and we need to get it right. 

We have a lot to cover today, so I again want to welcome our witnesses to today’s 
hearing, and I now yield to my good friend and colleague, Ranking Member Ralph 
Hall. 

Mr. Hall of Texas. Mr. Chairman, I thank you and today’s 
hearing on NASA’s National Aviation Operations Monitoring Serv- 
ice, NAOMS, is a timely hearing, especially considering the amount 
of scrutiny this program has received in the press. Several issues 
have arisen that bring into question the manner in which NASA 
closed out NAOMS, whether it achieved its original goals and the 
agency’s refusal to provide raw survey data to the press in response 
to a Freedom of Information Act request. I am optimistic that by 
the hearing’s conclusion and we hear these very capable men and 
women, if there are any on here, that we will have a clear under- 
standing regarding these and other pressing issues. 

And I do want to associate myself with NASA Administrator 
Mike Griffin’s public statement that lays out the agency’s philos- 
ophy on the treatment of research data. Like him, I believe NASA 
ought to be in the business of putting information in front of the 
public, not withholding it. That being said every care should be 
taken to protect the identities of survey respondents. NAOMS has 
promised pilots complete confidentiality to ensure their candid par- 
ticipation, and most folks believe that ought not to be breached. 

If information is disclosed that may allow respondents to be iden- 
tified, there will be a serious chilling effect in future survey efforts 
funded by the Federal Government, whether we are talking about 
pilots or other citizen groups who provide our government mean- 
ingful insight into a whole host of activities. In the case of NAOMS, 
we should be cognizant of striking a balance between transparency 
and confidentiality. 

I have the greatest faith in the Administrator. I have been 
through half a dozen or so administrators since I have been up 
here, and I think there is none surpasses him in background, abil- 
ity. He is a pilot, he is young, he is agile, and he is a lot of other 
things that are good for NASA. And I am just really proud of him 
and honored to have him come before this committee. 

NASA should release the data, but, you know, to help us all gain 
a better understanding of what it is telling us, they ought to pro- 
vide information, whether in the form of analysis, methodology, or 
reports, to give us a clear sense of context. But it is also important 
that the data be scrubbed, I think, to ensure errors are omitted. 
Get the errors out of there. 

I want to thank our witnesses for taking time from their busy 
schedules to appear before us this afternoon and acknowledge their 
hard work and preparation. All of us appreciate your willingness 
to be here, and Mr. Hall from Tennessee, we certainly well you and 
thank you, sir. 

I yield back my time. 

[The prepared statement of Mr. Hall of Texas follows:] 
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Prepahed Statement of Representative Ralph M. Hall 

Mr. Chairman, today’s hearing on NASA’s National Aviation Operations Moni- 
toring Service (NAOMS) is timely, especially considering the amount of scrutiny this 
program has received in the press. Several issues have arisen that bring into ques- 
tion the manner in which NASA closed out NAOMS, whether it achieved its original 
goals, and the agency’s refusal to provide raw survey data to the press in response 
to a Freedom of Information Act request. I am optimistic that, by the hearing’s con- 
clusion, we’ll all have a clear understanding regarding these and other pressing 
issues. 

I do want to associate myself with NASA Administrator Mike Griffin’s public 
statement that lays out the agency’s philosophy on the treatment of research data. 
Like him, I believe NASA ought to be in the business of putting information in front 
of the public, not withholding it. That being said every care should be taken to pro- 
tect the identities of survey respondents. NAOMS promised pilots complete confiden- 
tiality to ensure their candid participation, and that ought not be breached. If infor- 
mation is disclosed that may allow respondents to be identified, there will be a seri- 
ous chilling effect in future survey efforts funded by the Federal Government, 
whether we’re talking about pilots or other citizen groups who provide our govern- 
ment meaningful insight into a whole host of activities. In the case of NAOMS, we 
should be cognizant of striking a balance between transparency and confidentiality. 

NASA should release the survey data, but to help all of us gain a better under- 
standing of what it is telling us, they should also provide information, whether in 
the form of analysis, methodology, or reports, to give us a clear sense of context. 
It’s also important that the data be scrubbed to ensure errors are eliminated. 

I want to thank our witnesses for taking time from their busy schedules to appear 
before us this afternoon, and acknowledge their hard work and preparation. All of 
us appreciate your willingness to be here. 

Thank you, Mr. Chairman. 

Chairman Gordon. Thank you, Mr. Hall from Texas. 

If there additional Members who wish to submit additional open- 
ing statements, your statements will be added to the record. With- 
out objection, so ordered. 

[The prepared statement of Mr. Udall follows:] 

Prepared Statement of Chairman Mark Udall 

Good afternoon. I am disappointed that we have had to convene today’s hearing. 
But NASA’s stated rationale for refusing to release publicly information from the 
taxpayer-funded National Aviation Operations Monitoring Service (NAOMS) avia- 
tion safety survey is unsupportable and required concessional scrutiny. The safety 
of the public has to be our first priority, especially with more and more Americans 
flying every year. 

Specifically, in its response to the Associated Press’s request for release of the 
NAOMS aviation safety survey data, NASA stated that: “Release of the requested 
data, which are sensitive and safety -related could materially affect the public con- 
fidence in, and the commercial welfare of, the air carriers and general aviation com- 
panies whose pilots participated in the survey.” 

NASA’s response in effect seems to be saying that it sees its job as putting the 
commercial interests of the aviation industry above the public’s right to aviation 
safety information. 

That response is unacceptable. It’s certainly not in accordance with the National 
Aeronautics and Space Act of 1958, which created NASA and established objectives 
for the agency — one of which is “the improvement of the usefulness performance, 
speed, safety, and efficiency of aeronautical and space vehicles,” while directing 
NASA to operate in a manner that will “provide for the widest practicable and ap- 
propriate dissemination of information concerning its activities and the results there- 
of.” 

The NASA Administrator has since distanced himself from the language in 
NASA’s response to the FOIA request, saying that he regrets “the impression that 
NASA was in any way trying to put commercial interests ahead of public safety. That 
was not and will never be the case.” 

I’d like to hear the Administrator reiterate that stance at today’s hearing. And 
although I am glad that he has now agreed to release at least some of the NAOMS 
data publicly so that it can be used to help maintain and hopefully improve the safe- 
ty of the Nation’s airways, I feel strongly that all the NAOMS data should be made 
publicly available as soon as possible. 
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I intend to be vigilant to ensure that this release actually occurs in a timely man- 
ner. 

Former National Traffic Safety Board Chairman Jim Hall, who is one of our wit- 
nesses today, got it right in his prepared testimony when he wrote that “It is dif- 
ficult to overemphasize the importance of transparency and accountability in avia- 
tion. It is the single greatest reason why you are so safe when you get on an airplane 
today." I wholeheartedly agree. We need to work hard to expand that transparency 
and accountability — not restrict it. And that is why all the information from the 
study must be released — and soon. 

Yet, the struggle over the fate of the NAOMS data is not the only issue that needs 
attention at today’s hearing. We also need to decide where we should go from here. 
We will hear from a number of witnesses here today about the value of a com- 
prehensive, ongoing survey and analysis approach to aviation safety trend analysis 
and accident precursor identification — the approach exemplified by the NAOMS 
project. 

As Chairman of the Space and Aeronautics Subcommittee, I have oversight re- 
sponsibility for both NASA’s aeronautics and aviation R&D programs and FAA’s 
aviation R&D programs. 

I intend to make sure that the government is taking all necessary steps to have 
the aviation safety data sources and analysis tools that will be needed to maintain 
air safety in the coming years. 

Based on testimony we will hear today, there appears to be a great deal of merit 
to the NAOMS approach, and we need to assess whether NASA and FAA should 
reinstitute the project. Given its potential value and the modest amounts of funding 
required to make effective use of the NAOMS methodology relative to the more than 
$30 billion spent on NASA and FAA annually, I think the burden of proof should 
be on those who want to walk away from the investment made to date in the 
NAOMS project. 

I am aware that a number of FAA officials have indicated that the FAA is not 
interested in NAOMS and would rather develop a new aviation safety information 
system combining data from multiple existing safety and performance data bases. 
Making as effective use as possible of existing data bases is a worthy objective, and 
one that quite frankly FAA should have been doing all along. However, FAA’s own 
documentation states that it doesn’t envision completing more than “the Phase 1 
pre-implementation activities, including concept definition” for the proposed new 
combined Aviation Safety Information Analysis and Sharing (ASIAS) system until 
2013 at the earliest. 

That’s an unacceptably long time to wait, when it appears that NASA and FAA 
could he generating useful safety trend and accident precursor information — which 
will help keep the flying public safe — from a restarted NAOMS initiative almost im- 
mediately. 

It also doesn’t address the question of whether NAOMS could provide additional 
valuable insights into the safety status and trends for the Nation’s air transpor- 
tation system beyond those available from existing data bases. 

These issues go beyond what we are likely to have time to consider today, so I 
intend to have the Space and Aeronautics Subcommittee pursue them in the coming 
months. 

Mr. Chairman, we can take pride in the overall safety record of America’s air 
transportation system. However, we dare not rest on our laurels. We need to be vigi- 
lant to ensure that all is being done that should be done to maintain and improve 
that safety record — and the information gained from the taxpayer-funded NAOMS 
study is very important to our work. This hearing is an important step in meeting 
our safety oversight responsibilities, and I am glad we are holding it. 

[The prepared statement of Mr. Feeney follows:] 

Prepared Statement of Representative Tom Feeney 

When this hearing was first scheduled, allegations of cover up and document de- 
struction swirled in the air. So I initially thought — how did the Science and Tech- 
nolo^ Committee obtain jurisdiction over Sandy Berger’s escapades at the National 
Archives? Alas, that topic remains untouched. 

Originally, the Full Committee was to spend today examining the environmental 
and safety aspects of nanotechnology — a timely and thoughtful topic given 
nanotechnology’s current and future importance. Such a hearing would continue this 
committee’s serious treatment of serious issues. 

But like a cop on the beat, the powers-that-be have apparently given this com- 
mittee a quota of “oversight” tickets to write. Infractions must be found and high- 
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lighted with great drama. So the nanotechnology hearing was relegated to a sub- 
committee and replaced with today’s festivities. But to paraphrase Gertrude Stein, 
the trouble with today’s hearing is that “when you get there, there isn’t any there 
there.” 

Here’s today’s kerfuffle in a nutshell. Starting in fiscal year 1998, NASA funded 
a research project — the National Aviation Operations Monitoring Service 
(NAOMS) — that attempted to use telephone survey data to provide a representative 
picture of aviation system safety. Over eight years, $11.3 million (0.00867582 per- 
cent of NASA’s budget over this period) was spent on this non-peer reviewed re- 
search. 

Unfortunately, NAOMS failed to yield worthwhile information. Instead, it painted 
a picture of the aviation system with anomaly rates (such as engine failures) that 
bore no relationship with reality. It’s as if the public were polled and the data sug- 
gested a 75 percent approval rate for today’s Congress. Any politician would know 
that something was terribly wrong with that survey’s methodology. 

Four months ago, the Associated Press made a Freedom of Information Act (FOIA) 
request for the raw and rather stale data collected in the NAOMS study. NASA de- 
nied that request and used some inarticulate reasoning. 

When this matter was brought to NASA Administrator Mike Griffin’s attention, 
he promptly responded with his characteristic pattern of integrity, candor, and ac- 
tion. Griffin has vowed to bring openness and transparency to NASA. In that type 
of environment, participants feel empowered to acknowledge and address prob- 
lems — a behavior that could have averted the Challenger and Columbia tragedies. 
Thus, Griffin promptly acknowledged that NASA should have better handled this 
FOIA request and vowed to correct the matter. 

And so he has. Griffin has determined that this data should be released and will 
do so once confidential information is redacted (survey participants were promised 
confidentiality in return for their candor). Furthermore, he has cautioned about 
properly interpreting the data since the survey methodology appears to be quite 
flawed. 

In the wake of the Columbia Accident Investigation Board’s finding of a NASA 
culture discouraging openness and frankness, one would think Administrator Griffin 
would be commended for his leadership. After all, leaders set examples. Here he has 
promptly responded to a concern, acknowledged an error, and outlined corrective ac- 
tions. Isn’t this the type of conduct to be encouraged? 

But that would deviate from today’s script and ruin the planned drama. So like 
the abusive spouse who enjoys publicly brow-beating his partner, the Majority will 
undoubtedly pummel NASA’s finest Administrator in recent memory. No acknowl- 
edgement of error or corrective action will satisfy the belittling and rampaging 
spouse. 

Undoubtedly at another forum, today’s inquisitors will bemoan how skilled, ac- 
complished, and decent people eschew public service. Or how today’s Congress 
avoids addressing issues of genuine concern to the governed. 

[The prepared statement of Mr. Costello follows:] 

Prepared Statement of Representative Jerry F. Costello 

Mr. Chairman, I am pleased that the Committee is pursuing this issue, as the 
reports surrounding NASA’s NAOMS program and it’s refusal to release initial data 
have been troubling. 

As Chairman of the Aviation Subcommittee of the Committee on Transportation 
and Infrastructure, I firmly believe that safety must be our top priority. As Ranking 
Member of the Subcommittee last Congress, I called for a comprehensive hearing 
on aviation safety and since becoming Chairman, I have held numerous subsequent 
hearings that have highlighted the importance of this issue. 

What concerns me regarding NASA’s handling of the NAOMS study is that re- 
gardless of the initial findings of the study, this information has the ability to help 
improve transportation safety, and that should be our priority, not the possible ad- 
verse affects the information may or may not have on the industry. In addition, this 
situation has been handled poorly by NASA, and it fits into a pattern of reluctance 
to release information — particularly regarding safety — and concerns that NASA offi- 
cials are too close to, and too quick to protect, the interests of industry. 

Again, Mr. Chairman, I’d like to commend you for calling this hearing, I am very 
interested in learning the findings of this study, and how we can use the informa- 
tion to help ensure the safety of all air travelers. 

[The prepared statement of Mr. Miller follows:] 
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Prepared Statement of Representative Brad Miller 

The purpose of today’s hearing is to look at the National Aeronautics and Space 
Administration’s (NASA) management of the National Aviation Operations Moni- 
toring Service (NAOMS), and to examine how, in the absence of a system such as 
NAOMS, NASA plans on monitoring air safety in the future. 

Every year more planes are in the air, and each year brings new challenges to 
aviation safety. The purpose of NAOMS was to identify problems with both increas- 
ing demand and the introduction of new technologies. Instead of reacting to aviation 
disasters NAOMS would have been able to identify emerging safety problems. The 
program appears to be a cost-effective and scientifically valid way of looking at air- 
line safety. More important, I would like to know what NASA is going to do to en- 
sure American’s safety in the absence of NAOMS. 

I am glad that NASA and Administrator Griffin have voiced a willingness to re- 
lease the data gathered under the NAOMS project. Analysis of this data could be 
a key tool in understanding what is happening at US airports. I understand that 
there is some concern over the release of proprietary commercial data and the ano- 
nymity of survey participants. It is my strong hope that NASA will take realistic 
precautions to ensure anonymity, but not let that become an excuse not to release 
the data in a timely manner. 

[The prepared statement of Mr. Lipinski follows:] 

Prepared Statement of Representative Daniel Lipinski 

Thank you, Mr. Chairman. 

This is a very timely subject and one that is extremely important to the residents 
of the 3rd District of Illinois. Chicago is a key national and international aviation 
hub and collaboration is key to ensuring the continued safety and vitality of the 
aviation industry. At Midway International Airport in my District, working collabo- 
ratively we brought new safety upgrades online which will greatly enhance the safe- 
ty of the flying public and everyone who works at the airport. And through addi- 
tional collaboration, such as the sharing of informative data findings from your re- 
port, we can work to further improve the safety of our nation’s aviation industry. 

This issue hits especially close to home for me. Many remember the tragic acci- 
dent in 2005 when an aircraft skidded off the runway at Midway Airport into a 
passing car, killing a young boy. That is why, as a Member of the T&I Committee’s 
Subcommittee on Aviation, I worked hard to incorporate necessary funding into this 
year’s FAA reauthorization bill that will make our runways safer and increase avia- 
tion safety inspectors by more than one-third. I also sought to ensure the acceler- 
ated implementation of the Next Generation Air Transportation system, which will 
allow our air traffic control system to meet two to three times the amount of current 
demand and keep pace with the ever-increasing number of flights. 

[The prepared statement of Mr. Mitchell follows:] 

Prepared Statement of Representative Harry E. Mitchell 

Thank you, Mr. Chairman. 

Like most Americans, I was stunned last week to hear that NASA had refused 
to release the results of an $11 million survey of airline pilots on potential safety 
lapses in our nation’s aviation network. . .because the information “could under- 
mine public confidence in the airlines and could affect the airlines’ profits.” 

The idea that the Federal Government would put private profits ahead of the fly- 
ing public’s safety is as outrageous and inexcusable. 

The only thing more shocking about this awful decision is where it came from. 
We’re talking about NASA — the agency that houses some of the best and brightest 
minds on Earth. 

But it shouldn’t take a rocket scientist to figure out that safety comes first. 

Aviation is serious business in my district. One of the Nation’s largest airlines is 
headquartered in Tempe, and Phoenix Sky Harbor is now the eighth busiest in the 
country. We depend on aviation. . .and we depend on the Federal Government to 
keep our skies safe. 

NASA’s survey reportedly contains information. . .from pilots. . .about runway 
incursions, wildlife strikes, and near collisions. These are real risks. If pilots have 
concerns about them, we need to know. 

And if NASA wants to tell us that its survey methodology was flawed. . .and, 
therefore, the results of its survey are inconclusive. . .then we need to know how 
they were able to waste $11 million taxpayer dollars creating and conducting it. 
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Is it really asking too much for us to expect NASA to know a thing or two about 
scientific methodology? 

The flying public deserves an explanation. 

They deserve to know how this happened. . .but more importantly, what is being 
done to correct the situation, and what steps are being taken to ensure that some- 
thing like this never happens again. 

I look forward to hearing from our witnesses. 

I yield back. 

Chairman Gordon. At this time I would like to recognize our 
first panel. First we have Dr. Michael Griffin, who is the Adminis- 
trator of the National Aeronautics and Space Administration, and 
I will concur with Mr. Hall’s accolades, even the youthfulness. And 
we also have Mr. Jim Hall, who is a Managing Partner at Hall and 
Associates and is also the Former Chairman of the National Trans- 
portation and Safety Board. Welcome to you both. 

And Chairman Griffin, we will begin with you or Director Griffin. 

Panel 1: 

STATEMENT OF DR. MICHAEL D. GRIFFIN, ADMINISTRATOR, 

NATIONAL AERONAUTICS AND SPACE ADMINISTRATION 

(NASA) 

Dr. Griffin. Thank you, Mr. Gordon, Mr. Hall for your kind 
statements. I only wish I were still young, but, oh, well. It is all 
a matter of relativity here. Mr. Hall is my hero. He is still on the 
right side of the dais. 

So, thank you, Mr. Chairman, Members of the Committee for the 
opportunity to appear here today to discuss aviation safety and the 
NAOMS Project. When I was made aware last week that a NAOMS 
pilot survey data had been withheld under Freedom of Information 
Act request initiated by the AP, I asked Dr. Lisa Porter, our AA 
for Aeronautics Research, to investigate the matter. And I hope to 
provide you with the information that will address the questions 
and the concerns that have been raised by you and others in the 
past several days. 

Let me start by making three points clear up front. First, the 
survey results that we can legally release will be released. Period. 
Two, the contractor and NASA maintain master copies of all 
NA()MS survey results, and we have instructed the NAOMS 
project management team and the contractor, Battelle, to retain all 
records related to the project. Battelle provided the same direction 
to its subcontractors. Also, sir, your staff has this data. 

Three, the NAOMS Project had from its inception a planned and 
finite duration. It was not terminated early. It was, in fact, ex- 
tended, and it was not terminated early to provide funds for the 
Moon Mars Program or anything else. 

Quite simply, the NAOMS Project began in 1998, with the goal 
of developing methods to facilitate a data-driven approach to avia- 
tion systems safety analysis. To accomplish this goal required the 
generation of data that are statistically meaningful and representa- 
tive of the system. The NAOMS Project Team developed a survey 
methodology to acquire that data. The survey methodology develop- 
ment took about two years to complete. 

The actual data collection using that methodology began in April 
of ’01, and ended in December of ’04. During that time the project 
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team interviewed, surveyed approximately 24,000 commercial air- 
line pilots and 5,000 general aviation pilots. In early ’05, it was de- 
termined that the number of survey results collected were suffi- 
cient to evaluate whether the NAOMS survey methodology indeed 
produced statistically meaningful and representative data. 

NASA’s Aviation Safety and Security Program leadership then 
directed the NAOMS Project to complete the assessment of its sur- 
vey methodology and transfer it to industry and government deci- 
sion-makers and provided the FY 2005 funding to do that. 

It is worth noting that the 2004 review of NASA’s aerospace 
technology enterprise by the National Academies concluded at that 
time that there was not a compelling argument for continued inde- 
pendent data collection in the NAOMS Project. In fact, quoting 
from that report, the “NAOMS Project seems to be developing a 
methodology to establish trends in aviation safety performance that 
are already available through other sources within industry and 
government.” 

In 2006, the Aviation Safety Program of NASA’s Aeronautic Re- 
search Mission Directorate provided additional funding to complete 
the transition and to document the results. The transition of the 
survey methodology has now been successfully completed, but the 
documentation has taken longer to complete than anticipated. That 
will be completed by the end of this year. 

Now, it has been widely reported that NAOMS funding was cut 
or prematurely ended. That is not the case. When the project origi- 
nated in 1998, it was intended to continue until 2004, as indicated 
in project briefings that were provided to various government and 
industry audiences when it began. Copies of these briefings have 
been provided to Committee staff for the record. 

As I previously mentioned, funding was extended through ’06, to 
allow for transition of the methodology and final documentation. 
And the total amount that we have now spent on this effort has 
been $11.3 million. 

Now, with all that said, the arch, overarching goal of trying to 
develop methodologies the enabled data-driven safety analyses is 
one that we at NASA continue to embrace in the current Aviation 
Safety Program, and we do so in close partnership with the FAA, 
industry, and academia. 

In order to significantly reduce the accident rate to meet the ex- 
pected growth of the next generation air transportation system, it 
is imperative to develop a robust safety information system that 
discovers safety precursors before accidents occur. Accomplishing 
this requires the ability to combine and analyze enormous amounts 
of data from varied sources to detect and act on new safety threats. 

To address this challenge, NASA and FAA are combining their 
separate and unique skills and resources under clearly-defined 
roles and responsibilities. NASA is focused on the development of 
advanced analysis alga rhythms that can be implemented in a com- 
prehensive system that the FAA can utilize to effectively analyze 
a wide variety of safety data. 

In order to ensure that the technology is effectively transitioned 
between the organizations, a program plan has been developed and 
is being executed. The initial response to this approach from the 
stakeholder community has been very positive. The FAA’s Research 
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Engineering and Development Advisory Committee, the REDAC 
Safety Subcommittee, recently reported and recent means in Octo- 
ber of ’07, that it, “Believes significant progress has been made 
over the past year,” in defining the program and its execution. The 
Safety Subcommittee credited the leadership of both FAA and 
NASA for, “Driving a well-integrated plan that will form the basis 
for proactive risk identification and assessment in the future.” 

There has been a lot of speculation in the press regarding what 
the NAOMS survey might reveal about the safety of the National 
Aerospace System. Several briefings were given to other govern- 
ment agencies and industry organizations by members of the 
NAOMS Project Team, and some of those presentations included 
some analyses that were based upon extrapolating the survey re- 
sults to obtain, to estimate absolute numbers of events that would 
occur within a given time period. When this was done, for many 
of these events the numbers were significantly higher than re- 
ported by other means such as the Aviation Safety Reporting Sys- 
tem or ASRS that NASA manages by statute. 

However, no attempt was made to validate the NAOMS extrapo- 
lation methodology, and indeed, given the results for some cases 
such as engine failure events that are highly public and carefully 
documented affairs, there may be a reason to question the validity 
of the methodology itself. It is interesting to note here that in 
NASA’s own Safety Reporting System, the NSRS, 40 percent of the 
events which are reported are either found — are found later to be 
either overstated, unverifiable, or not significant enough to require 
follow-up. 

While some analysis of the survey results was presented to 
NASA, other government agencies and other personnel, unfortu- 
nately none of the research conducted in the NAOMS Project, in- 
cluding the underlying survey methodology, was peer reviewed or 
has been peer reviewed to date. Accordingly, any product of the 
NAOMS Project, including the survey methodology, the resulting 
data, and any analysis of that data should not be viewed or should 
not be considered at this stage as having been validated. 

So in plain speaking, when I said we can release whatever data 
can, we will release whatever data we can be legally released, and 
we will do that, we do not certify that data. There has been consid- 
erable attention in the press to the supposed destruction of 
NAOMS data. In fact, Battelle, the prime contractor, maintains 
master copies of all survey data on CDs and other back-up media 
in its Mountain View facility. NASA’s Ames Research Facility at 
Moffett Field also has copies of this data. 

We had directed Battelle to recover or to ensure the secure de- 
struction of any copies of survey results that might be held at loca- 
tions outside Mountain View. This includes copies held by present 
or past Battelle NAOMS subcontractors. The purpose of that re- 
quest was to ensure compliance with NASA’s data security require- 
ments as part of the contract close-out process, because the con- 
tract was scheduled to end in October of ’07. This request in no 
way jeopardized the security of the master copies, which remain se- 
cure at Battelle and at Ames. 

To ensure that no instruction — no destruction of survey results 
occurs, however, including those held by subcontractors, after the 
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concerns about data destruction were raised by this committee, 
NASA directed the NAOMS Project Management Team and 
Battelle to retain all records related to the NAOMS Project, and 
Battelle provided the same direction to its subcontractors. We have 
provided all this information to the Committee. 

Finally, let me focus on the Freedom of Information Act request. 
Under federal law we at NASA are required to protect confidential 
commercial information that is voluntarily provided to the agency 
and would not customarily be released to the public. That is the 
law. In preparing our response to the AP Freedom of Information 
Act appeal, the characterization of the requested data by Ames re- 
searchers raised concerns that the data likely contained confiden- 
tial commercial information. This characterization was the basis for 
withholding the data under Exemption 4. 

Now, considerable attention has been focused on one sentence in 
the final determination letter suggesting the data was being with- 
held because, “It could affect public confidence in and the commer- 
cial welfare of air carriers and general aviation companies.” Now, 
I have already made it clear that I do not agree with the way this 
was written, and I regret any impression that NASA was or would 
in any way try to put commercial interests ahead of public safety. 
That was not and will never be the case. 

As for our plans for the data, I have directed that all NAOMS 
data not containing confidential commercial information or infor- 
mation that could compromise the anonymity of individual pilots be 
released as soon as possible. But at present we are concerned that 
it might be possible that a knowledgeable person could identify a 
specific individual or reconstruct specific events back to a specific 
individual, and we must protect against that, and no proprietary 
commercial information could be compromised. 

We will receive a written report by Battelle by the end of this 
year that will include a description of the methodology, the ap- 
proach, the field trials, et cetera. We will make this report avail- 
able to any interested party. We intend to continue to emphasize 
the importance of peer review of all research results, whether con- 
ducted by NASA’s researchers or our contractors funded by NASA. 
Peer review is critical to the achievement of technical excellence. 

Let me conclude by thanking you for this opportunity to appear 
before you to discuss the NAOMS issue and to answer your ques- 
tions. Thank you. 

[The prepared statement of Dr. Griffin follows:] 

Prepared Statement of Michael D. Griffin 

Mr. Chairman and Members of the Committee, thank you for this opportunity to 
appear before you today to discuss the National Aviation Operations Monitoring 
Service (NAOMS) project, and the issue concerning the release of data obtained by 
various researchers pursuant to that project. When I was made aware last week 
that NAOMS pilot survey data had been withheld under a Freedom of Information 
Act request initiated by the Associated Press, I asked Dr. Lisa Porter, Associate Ad- 
ministrator for Aeronautics Research, to investigate the matter. I hope to provide 
you with information that will address the questions and concerns that have been 
raised by you and others during the past several days. 

What is NAOMS? 

There has been some confusion regarding what NAOMS actually is. The NAOMS 
project began in 1998 with an overarching goal of developing methods to facilitate 
a data-driven approach to aviation system safety analysis. Accomplishing this goal 
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requires the generation of data that are statistically meaningful and representative 
of the system. The NAOMS project team decided to develop a survey methodology 
to acquire such data. The survey methodology development took roughly two years 
to complete. The actual data collection using the methodology began in April 2001 
and ended in December 2004. During that time, the project team surveyed approxi- 
mately 24,000 commercial airline pilots and approximately 5,000 general aviation 
pilots. 

In early 2005, it was determined that the amount of data collected was sufficient 
to evaluate whether the NAOMS survey methodology indeed produced statistically 
meaningful and representative data. NASA’s Aviation Safety and Security Program 
leadership thus directed the NAOMS project to complete the assessment of its sur- 
vey methodology and transfer it to industry-government decision-makers (Commer- 
cial Aviation Safety Team [CAST] and Air Line Pilots Association [ALPA]), and pro- 
vided FY 2005 funding to do so. It is worth noting that the 2004 Review of NASA’s 
Aerospace Technology Enterprise by the National Academies concluded that there 
was not a compelling argument for continued independent data collection in the 
NAOMS project. In FY 2006, the Aviation Safety Program of the Aeronautics Re- 
search Mission Directorate (ARMD) provided additional funding to complete the 
transition and to document the results. The transition of the survey methodology 
has been successfully completed, but the documentation has taken longer to com- 
plete than anticipated. The documentation will be completed by the end of this year. 

Why was funding for NAOMS cut? 

It has been widely reported that NAOMS funding was cut or prematurely shut 
down. That is not the case. When the project originated in 1998, it was intended 
to continue until 2004, as indicated in project briefings that were provided to var- 
ious government and industry audiences when the project began. (These briefings 
have been provided to the Committee for the record. Later briefings indicated an 
extension to 2005.) As I previously mentioned, funding was extended through 2006 
to allow for transition of the methodology and final documentation. The total 
amount we spent on this effort was $11. 3M. 

That said, the overarching goal of trying to develop methodologies that enable 
data-driven system safety analyses is one that NASA continues to embrace in its 
current Aviation Safety Program, in close partnership with the FAA, industry, and 
academia. In order to continually and significantly reduce the accident rate to meet 
the expected growth of the Next Generation Air Transportation System (NextGen), 
it is imperative to develop a robust safety information system that discovers safety 
precursors before accidents occur. Accomplishing this requires the ability to combine 
and analyze vast amounts of data from many varied sources to detect and act on 
new safety threats. 

NASA and the FAA are combining their unique skills and resources under clearly 
defined roles and responsibilities to address this challenge. NASA is focused on the 
development of advanced analysis algorithms that can be implemented in a com- 
prehensive system that the FAA can utilize to effectively analyze a wide variety of 
safety data. In order to ensure that the technology is effectively transitioned be- 
tween organizations, a program plan has been developed and is being executed. The 
initial response to this approach from the stakeholder community has been very 
positive. The FAA Research Engineering and Development Advisory Committee 
(REDAC) Safety Subcommittee recently reported out to the REDAC in October 2007 
that it “believes significant progress has been made over the past year” in defining 
the program and its execution. The Subcommittee credited the leadership of both 
the FAA and NASA for “driving a well integrated plan that will form the basis for 
proactive risk identification and assessment in the future.” 

What do the data show? 

There has been much speculation in the press regarding what the data will reveal 
about the safety of our national airspace system. Several briefings were given to 
other government and industry organizations by members of the NAOMS project 
team, and some of those presentations included some analyses that were based upon 
extrapolation methods to estimate absolute numbers of events occurring within a 
given time period. For many of these events, the numbers were significantly higher 
than reported by other means, such as the Aviation Safety Reporting System 
(ASRS). However, there was no attempt made to validate the extrapolation method- 
ology. Indeed, given the results for some examples such as engine failure events, 
there may be reason to question the validity of the methodology. 

While some analysis of the data was presented to NASA and other government 
personnel, unfortunately, none of the research conducted in the NAOMS project, in- 
cluding the survey methodology, has been peer-reviewed to date. Accordingly, any 
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product of the NAOMS project, including the survey methodology, the data, and any 
analysis of that data, should not be viewed or considered at this stage as having 
been validated. 

Did NASA destroy any data? 

There has been considerable attention in the press to the supposed destruction 
of NAOMS data. Battelle Memorial Institute, the prime contractor, maintains mas- 
ter copies of all NAOMS survey results on compact discs and other backup media 
in its Mountain View, Calif, facility. NASA’s Ames Research Facility at Moffett 
Field, Calif, also maintains copies of this data. 

NASA had directed Battelle to recover, or ensure secure destruction of, any copies 
of the NAOMS data that might be held at locations outside of Mountain View. This 
includes copies held by present or past Battelle NAOMS subcontractors. The pur- 
pose of this request was to ensure compliance with NASA data security require- 
ments as part of the contract close-out process, because the contract is scheduled 
to end in October 2007. This request in no way jeopardized the security of the mas- 
ter copies, which remain secure at Battelle and the Ames Research Facility. 

To ensure that no destruction of data, including data held by sub-contractors, oc- 
curred after concerns about data destruction were raised by this committee, NASA 
notified the NAOMS project management team and Battelle to retain all records re- 
lated to the NAOMS project. Battelle provided the same direction to its subcontrac- 
tors. 

Dissemination of research results 

One of the most important NASA principles is to ensure the dissemination of re- 
search results to the widest practical and appropriate extent. This principle has re- 
ceived particular focus during the restructuring of ARMD. The emphasis on open 
dissemination is clearly stated in ARMD’s fully and openly competed NASA Re- 
search Announcements as well as in the Space Act Agreements that it establishes 
with commercial organizations for collaborative research. Furthermore, all of 
ARMD’s project plans include documentation and publication of results as 
deliverables. We firmly believe in the importance of the peer-review process, which 
is essential for ensuring technical excellence. 

Why did NASA reject the FOIA request? 

Under federal law, NASA is required to protect confidential commercial informa- 
tion that is voluntarily provided to the agency and would not customarily be re- 
leased to the public. In preparing the response to the Associated Press’ Freedom of 
Information Act appeal, the characterization of the requested data by Ames re- 
searchers raised concerns that the data likely contained confidential commercial in- 
formation. This characterization was the basis for withholding the data under Ex- 
emption 4. 

Considerable attention has been focused on one sentence in the final determina- 
tion letter suggesting the data was being withheld because it could “affect the public 
confidence in, and the commercial welfare of, the air carriers and general aviation 
companies.” I have already made clear that I do not agree with the way it was writ- 
ten. I regret any impression that NASA was in any way trying to put commercial 
interests ahead of public safety. That was not and never will be the case. 

NASA plans 

I have directed that all NAOMS data that does not contain confidential commer- 
cial information, or information that could compromise the anon3Tnity of individual 
pilots, be released as soon as possible. The release of this data will be accompanied 
with the proviso that neither the methodology nor the results have received the 
level of peer review required of a NASA research project. Therefore, the survey 
methodology and the data should not be considered to have been verified. 

NASA will receive a final report from Battelle by December 31, 2007 that will in- 
clude a comprehensive description of the methodology, including approach, field 
trials, etc. NASA will make this report available to any interested party. 

We intend to continue to emphasize the importance of peer-review of all research 
results, whether conducted by NASA researchers or contractors funded by NASA. 
Peer-review is critical to the achievement of technical excellence. 

Concluding remarks 

Let me conclude by thanking you again for this opportunity to appear before you 
to discuss NAOMS and to answer your questions. 

Chairman Gordon. Thank you, Dr. Griffin, for your candor once 
again, and Mr. Hall, you are recognized. 
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STATEMENT OF MR. JAMES E. HALL, MANAGING PARTNER, 

HALL AND ASSOCIATES, LLC; FORMER CHAIRMAN, NA- 
TIONAL TRANSPORTATION SAFETY BOARD (NTSB) 

Mr. Hall. Thank you, Mr. Chairman, Representative Hall, and 
distinguished Members of this committee. I have provided extended 
testimony that I would like to submit for the record if it pleases 
the Chairman. 

Chairman Gordon. No objection. 

Mr. Hall. And it is I think significant that this meeting is being 
held on the eighth anniversary of the Egypt air accident that oc- 
curred during my watch at the NTSB. I appreciate the opportunity 
to speak on aviation safety. Can NASA do more to protect the pub- 
lic? This is one of the issues that was addressed 10 years ago by 
the 1996, White House Commission on Aviation Safety and Secu- 
rity, which I had the privilege to serve on. The commission was 
prompted in large part by the tragic aviation accidents of that year, 
ValuJet and TWA 800. 

Before I begin, however, I would like to share with this com- 
mittee that the most important thing, the most important thing I 
learned in my seven years at the NTSB, and that is the culture of 
aviation safety has been built upon constant critical self-examina- 
tion. Open and transparent information flow is the key to aviation 
safety. With openness in mind, the members of the 1996 commis- 
sion felt that we needed to get ahead of events in a rapidly chang- 
ing environment to be able to improve the safety and security of 
aviation before, not after, another tragic accident occurred. 

Notable safety recommendations issued by the commission in- 
cluded the establishment of standards for continuous safety im- 
provement, a target rate of 80 percent was said for the reduction 
of fatal accidents. And we continued, which has considerable exper- 
tise in resources and the area of safety research, to expand its in- 
volvement in the promotion of aviation safety. 

In this last point the extremely important safety research func- 
tion is what brings us here today. Since the commission met, we 
have seen a 65 percent reduction in fatal accidents. While this is 
certainly welcome news, there are dangerous trends in the aviation 
industry that stand to jeopardize that progress. These include air 
traffic controller and pilot staffing levels, the number of runway in- 
cursions, the dramatic increase we will see in general aviation, the 
development and implementation of NextGen, UAVs and the explo- 
sion in passenger levels, which the Chairman referred to and which 
is estimated to reach 2.3 billion by the year 2027. 

More work indeed remains, which makes it all the more frus- 
trating that NASA withheld results obtained from what I first be- 
lieved was an $8.5 million taxpayer-funded National survey of al- 
most 24,000 pilots. This survey reportedly states that runway in- 
cursions, wildlife strikes, and near collisions occur at a rate at least 
twice as much as is commonly thought. 

As justification to its denial of a FOIA request the NASA spokes- 
man cited the potentially harmful effects on the commercial welfare 
of the air carriers and public confidence in aviation. 

Such action, I believe, runs counter to the safety culture men- 
tality that the government and industry have worked to create over 
the past 10 years. As the Government Accounting Office has oh- 
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served, transparency forms the fundamental basis for any safety 
program. If we don’t know something is broken, we cannot fix it. 

It is difficult to overemphasize the importance of transparency 
and accountability in aviation. I know each one of you Members fly 
probably weekly. I believe that that transparency and account- 
ability that is the single greatest reason you are so safe when you 
get on an airplane today. The history of transparency began with 
the Wright Brothers, who assisted in the investigation of the first 
fatal aviation accident in 1908, and used the results to incorporate 
changes to their flying machine in order to save lives. 

This open process has resulted in numerous important advances 
in aviation. NTSB investigations and recommendations have led to 
the advent of the Traffic Alert and Collision Avoidance System, 
commonly known as TACAS, Low-Level Wind Sheer Alert System, 
anti-collision Systems and Ground Proximity Warning Systems to 
name but a few. 

To repeat, information flow is the key to safety. In its investiga- 
tion into the two Shuttle accidents in 1986, and 2003, NASA itself 
noted that a decline in transparency and accountability among 
management and not simply a lack of adequate funding for safety 
was a root cause of both disasters. 

Furthermore, because major aviation accidents are now such a 
rarity, our ability to identify risks and maintain or increase safety 
now depends primarily on our ability to fully analyze incidents and 
trends. A true safety culture requires transparency and constant 
diligence. The vigilance, excuse me, is required of all involved in 
the aviation industry, but its absence is probably most glaring 
when it is the fault of government, the servants of the American 
people. 

NASA needs to release this information and fulfill its responsibil- 
ities as envisioned by the 1996, White House Commission. To do 
otherwise, I believe, flies in the face of aviation history, responsible 
government, and common sense. 

Thank you, Mr. Chairman. 

[The prepared statement of Mr. Hall follows:] 

Prepared Statement of James E. Hall 
Good afternoon Mr. Chairman and Members of the Committee: 

Thank you for allowing me the opportunity today to speak on the subject of Avia- 
tion Safety: Can NASA Do More to Protect the Public'^ My name is Jim Hall, and 
for more than seven years I served as Chairman of the National Transportation 
Safety Board (NTSB). I also had the honor to serve as a Commissioner on the 1996 
White House Commission on Aviation Safety and Security. 

As you know, the NTSB is an independent federal agency charged by Congress 
with investigating every civil aviation accident in the United States as well as sig- 
nificant accidents in the other modes of transportation — railroad, highway, marine, 
and pipeline. Since its inception in 1967, the NTSB has investigated more than 
124,000 aviation accidents and over 10,000 surface transportation accidents, and 
has also assisted many foreign governments with their own investigations. In its 
issuance of more than 12,000 recommendations in all transportation modes to more 
than 2,200 recipients, the Board has established a solid reputation for diligence and 
impartiality. From 1994 to 2001, I headed this organization that serves as the “eyes 
and ears” of the American people at aviation and other transportation accidents 
across the country and around the world. Now, as a transportation safety and secu- 
rity consultant, I continue my commitment to promoting safety in our nation’s 
transportation system. 

Today I would like to put the current aviation safety environment in a historical 
context. Ten years ago we were confronted with a special situation of change and 
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risk in the aviation industry. In response, the Commission on Aviation Safety and 
Security was formed, which I will discuss in a moment. I believe that today we face 
a similar situation, what I like to call “the next generation of risks.” 

The Gore Commission 

In 1996, the Federal Government initiated a decade-long overhaul of aviation 
safety that began with the establishment of the White House Commission on Avia- 
tion Safety and Security, headed by Vice President A1 Gore. The Gore Commission, 
as it would come to be called, was formed for three major reasons. 

On May 11, 1996, ValuJet flight 592 crashed in the Everglades after an in-flight 
fire caused by transported oxygen canisters, killing all 110 people on board. In the 
resulting NTSB investigation, we found airline contractors and ValuJet — an airline 
that had been formed just three years prior to the flight 592 crash — negligent in 
several areas, including oversight and mishandling of hazardous materials. We also 
determined if previous recommendations issued in 1988 regarding fire detection and 
extinguishing systems had been adopted, flight 592 would likely not have crashed. 
It was, therefore, a largely preventable and tragic loss of life. 

The second major reason for the formation of the Gore Commission was an inci- 
dent occurring only two months after the ValuJet crash. On July 17, 1996, Trans 
World Airlines Flight 800 experienced an in-flight break up following an explosion 
of the center wing fuel tank (CWT) shortly after take off from John F. Kennedy Air- 
port in New York City, killing all 230 people on-board. After an extensive 17-month 
investigation, we determined the source of the explosion to be an ignition of the 
flammable fuel/air mixture in the tank, an ignition most likely caused by a short 
circuit outside of the fuel tank. The NTSB issued specific recommendations on wir- 
ing and design as well as broader management of the aging aircraft fleet. In the 
period immediately following the crash, concerns of possible security problems led 
President Clinton to call for an immediate report on aviation security within 45 
days. 

The third reason that led to the Gore Commission was the general feeling that 
aviation — an industry that generated $300 billion annually and employed close to 
one million Americans — was undergoing profound changes. In the ten years prior to 
1996, the Federal Aviation Administration (FAA) had certified twenty new aircraft 
models and the number of passengers fl3dng in the United States exceeded more 
than a half billion. New digital technology was being developed to improve commu- 
nication and navigation. Sixty new airlines, such as ValuJet, had started operations 
since 1992. The commercial airline fleet was both quickly aging and in the midst 
of rapid replacement of aircraft. The domestic market faced the possibility of in- 
creased competition from foreign carriers. To add to this, the FAA predicted that 
by 2007, more than 800 million passengers would fly in the United States. 

In this setting, and in light of two very public and tragic accidents, the Gore Com- 
mission was created with three specific mandates: to examine security threats and 
ways to address them; to analyze overall changes in the industry and the appro- 
priate adaptation of government regulation to these changes; and to look at techno- 
logical changes in the air traffic control system. All of us involved at the time felt 
that we needed to “get ahead” of events in a rapidly changing environment, to im- 
prove the safety and security of aviation before — not after — another tragic accident 
occurred. 

Over six months I and the fellow members of the commission — which included the 
Secretary of Transportation, two retired Air Force generals, the director of FBI, and 
several scientists — conducted dozens of site visits in the U.S. and abroad, held six 
public meetings, and co-sponsored an International Conference on Aviation Safety 
and Security attended by over 700 representatives from sixty-one countries. From 
our findings we issued some fifty-one separate recommendations covering a variety 
of issues from safety to security to the notification of family members following an 
incident. 

Notable safety recommendations issued by the Commission included: the estab- 
lishment of standards for continuous safety improvement (a target rate of 80 per- 
cent was set for the reduction of fatal accidents); extension of FAA oversight to avia- 
tion contractors; the simplification of Federal Aviation Regulations; an emphasis on 
human factor safety research and training; and an extension of whistleblower statu- 
tory protection to the aviation industry. To be sure, not every recommendation made 
was subsequently enacted, nor was every possible safety item individually ad- 
dressed — no commission can claim perfection in this respect. Nevertheless, many 
recommendations were in fact adopted and perhaps even more significantly, the 
Presidential attention shown to the issue sent a message to both government and 
industry leaders that the establishment of a safety culture was not an option. It is 
therefore no coincidence that in the ten year period following the commission, the 
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industry successfully reduced fatal accidents by 65 percent, 15 percent shy of the 
national goal, but noteworthy nonetheless. 

This reduction was due not only to the actions of the airlines but to government 
efforts as well. The Commission charged the FAA, Department of Transportation 
(DOT), and NTSB to be more vigorous in their certification, regulation, and inves- 
tigative functions. It also urged the expansion of research, and specifically noted the 
need for the National Aeronautics and Space Administration (NASA), “which has 
considerable expertise and resources in the area of safety research, to expand its 
involvement in the promotion of aviation safety.” 

As a result of the Commission’s recommendation, NASA launched its $500 million 
Aviation Safety Program (AvSP) a partnership with the Department of Defense 
(DOD), FAA, and the aviation industry to focus on accident prevention, accident 
mitigation, and aviation system monitoring and modeling. It is this last point, the 
extremely important safety research function, which brings us here today. Given a 
rapidly changing environment and a new set of risks, the attempt on the part of 
NASA to suppress safety data is a grave and dangerous challenge to the safety cul- 
ture that has developed over the last century of aviation history, due to lessons 
learned from past accidents and incidents. 

The Next Generation of Risks 

The 65 percent reduction in fatal accidents over the past ten years is certainly 
welcome news, but while many advances have been made, there are dangerous 
trends in the aviation industry that stand to jeopardize this progress. 

We are currently in the middle of an air traffic controller staffing crisis. Fueled 
in part by the lack of a contract, this crisis has industry-wide consequences includ- 
ing: more and longer flight delays, combined radar and tower control positions, and 
an increased use of mandatory overtime resulting in an exhausted, stressed out, and 
burned out workforce. According to the National Air Traffic Controller Association 
(NATCA) there were 856 retirements in fiscal year 2007, (7.4 percent of the total 
experienced controller workforce), leaving the country with a 15-year low in the 
number of fully certified controllers and a surplus of new hires — many with no air 
traffic control experience or education. Total controller attrition in FY07 was 1,558, 
nearly wiping out any net gains in total staffing made by the FAA’s hiring efforts. 
In fact, the agency estimates it will lose about 70 percent of the air traffic controller 
workforce over the next 10 years. 

Air Traffic Controllers are not the only ones retiring. Pilot staffing levels are dan- 
gerously low as a result of retiring baby-boomers and an explosion of new airlines 
and increased airline fleets in Asia and the Middle East, raising similar concerns 
of an influx of inexperienced and insufficiently trained pilots. In 2009, airlines will 
have to fill 20,000 openings due to retirements and other factors. Some airlines fac- 
ing pilot shortages are lowering experience requirements to the FAA minimum. 

Other operational and technological areas present potentially problematic trends 
as well. Runway incursions, which have been on the NTSB’s Most Wanted Safety 
Improvement list since 2001, totaled over 1,300 between fiscal years 2003 and 2006. 
Among the aviation safety community, the Tenerife incursion accident that killed 
583 people in the Canary Islands in 1977 stands as a sober reminder of the impor- 
tance of getting this number down. The April 25, 2006 crash of an unmanned aerial 
vehicle (UAV) in Nogales, Arizona, and the resulting NTSB investigation and 22 
recommendations illustrate the potential problems with the growing expansion of 
drone flights in the U.S. General aviation and the air ambulance fleet have also in- 
creased in the last ten years; however the FAA does not collect actual flight activity 
data for general aviation operators and air teixis, instead using an annual survey 
to query a sample of registered aircraft owners. 

Several new aircraft types will emerge in the years ahead, ranging from the 
jumbo Airbus A380 that seats more than 500 passengers — a jet so large as to raise 
safety concerns in its own right — to very light jets that might transport six or fewer 
passengers. As many as four to five hundred new very light jets are scheduled to 
be introduced into American airspace each year starting in 2008. 

The Next Generation Air Transportation System (NextGen), a major and much- 
needed technology upgrade for the air traffic control system scheduled for comple- 
tion in 2025, will only add to the variables that need to be factored in aviation safe- 
ty, especially if NextGen is not adequately funded, implemented, or regulated. 

Overshadowing all these developments is a major growth in demand for air travel. 
In fiscal year 2006, over 740 million passengers flew in American skies. That figure 
is projected to reach one billion by 2015 and close to 2.3 billion by 2027. These num- 
bers are absolutely staggering. On January 1, 2007 federal regulations on the quan- 
tity of planes able to use J.F.K. airport ended, and traffic has increased by some 
20 percent. Congestion and resulting delays may be inconvenient, but it also in- 
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creases the potential for mishaps. As a Government Accounting Office (GAO) report 
released in February of this year noted, “although the system remains extraor- 
dinarily safe, if the current accident rate continues while air traffic potentially tri- 
ples in the next 20 years, this country would see nine fatal commercial accidents 
each year, on average.” 

I am not suggesting that nothing is being done to address these issues. I think 
individuals such as Marion Blakely, former administrator of the FAA, and Bobby 
Sturgell, current Acting Administrator of the FAA, have taken strong steps to ad- 
dress safety concerns. And yet, to again cite the GAO study, “FAA’s approaches to 
safety require that the agency obtain accurate and complete data to monitor safety 
trends, fully implement its safety programs, and assess their effectiveness to deter- 
mine if they are focused on the greatest safety risk. FAA has made progress in this 
area but more work remains [italics added].” 

The Withholding of NASA’s Data 

More work indeed remains, which makes it all the more frustrating that NASA 
withheld results obtained from an $8.5 million teix payer funded national survey of 
almost 24,000 pilots. This survey reportedly states that runway incursions, wildlife 
strikes, and near collisions occur at a rate at least twice as much as is commonly 
thought. As justification to its denial of a Freedom of Information Act request, 
NASA cited the potentially harmful affects on the commercial welfare of the air car- 
riers and general aviation companies. 

Such an action runs exactly counter to the safety culture mentality the govern- 
ment and industry have worked to create over the past ten years. As the GAO ob- 
served, transparency forms the fundamental basis for any safety program. If we 
don’t know something is broken, we cannot fix it. If we do not know that runway 
incursions are actually occurring at a much higher level, then we cannot take steps 
and assign the resources to deal with them. 

It is difficult to overemphasize the importance of transparency and accountability 
in aviation. It is the single greatest reason why you are so safe when you get on 
an airplane today. The history of transparency began with the Wright Brothers, who 
assisted in the investigation of the first fatal aviation accident and used the results 
to incorporate changes to their flying machine in order to save lives. In September 
1908, five years after the Wrights’ historic flight, Orville and Lt. Thomas Selfridge 
were conducting an aerial demonstration for the Army in Fort Meyers, Virginia 
when their airplane stopped responding to controls and crashed, injuring Orville 
and killing Lt. Selfridge. The Wright Brothers’ commitment to objective scrutiny and 
constant improvement set an historic precedent and has led to a safety culture in 
aviation that is built on fact finding, analysis and open sharing of information to 
advance aviation and save lives. This open process has resulted in numerous impor- 
tant advances in aviation. In the modern era, NTSB investigations and rec- 
ommendations have led to smoke detectors in airplane lavatories, floor level lighting 
strips to lead passengers to emergency exits, anti-collision systems, and ground 
proximity warning devices, to name but a few. 

The industry often very clearly responds to the efforts of safety research even be- 
fore investigations are completed. On September 8, 1994, USAir flight 427, a Boeing 
737, crashed while on approach to Pittsburgh, Pennsylvania. After 80,000 hours of 
investigation, the NTSB had not yet completed its final report but had issued sev- 
eral recommendations. In response, Boeing and the FAA began developing and certi- 
fying several modifications to the 737 main rudder power control unit (PCU) servo 
valve. The FAA proposed an Airworthiness Directive to require the installation of 
newly designed PCUs within two years. Most airlines began providing training to 
pilots on the recognition, prevention, and recovery of aircraft attitudes normally not 
associated with air carrier flight operations. 

On October 31, 1994, an American Eagle ATR-72 crashed in Roselawn, Indiana. 
Seven days after the crash of an ATR-72 in Roselawn, Indiana, we issued rec- 
ommendations covering the operation of those aircraft in icing conditions. Thanks 
to a then state-of-the-art flight recorder, we were able to learn within days that the 
French-built ATRs upset was initiated by a rapid deflection of the right aileron. The 
NTSB deduced that this deflection was caused by the accumulation of a substantial 
amount of ice on the wings during the 30 minutes the plane was in a holding pat- 
tern. Within a week of the accident, the NTSB issued urgent safety recommenda- 
tions to the FAA to restrict the operation of ATRs in icing conditions until a fix 
could be developed to counteract the phenomenon the accident aircraft encountered. 
Within a month, following test flights in the United States and France, the FAA 
effectively grounded the aircraft in icing conditions. A redesign of the wing anti- 
icing boots was developed, and the modified airplanes returned to the skies. 
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One of the keys to the Roselawn investigation ■was the fact that the flight data 
recorder (FDR) was recovered and that it recorded some 98 parameters, giving in- 
vestigators ample information with which they could quickly establish the cause of 
the accident and the most appropriate fix. This contrasts with the FDR on-hoard 
flight 427 the previous month, which recorded only 11 parameters and in so small 
part delayed the release of the final investigation report hy over four years. In a 
sense, NASA’s refusal to release their safety data is tantamount to denying inves- 
tigators access to hlack boxes. Both actions seriously impede the ability to determine 
potentially critical safety concerns. 

Information flow is the key to safety, whether to the investigator actually assem- 
bling pieces on the ground or to the analyst compiling survey data back in the office. 
In its investigations into the two Shuttle accidents in 1986 and 2003, NASA itself 
noted that a decline in transparency and accountability among management — and 
not simply a lack of adequate funding for safety — was a root cause of both incidents. 

The investigation into the Challenger explosion specifically faulted management 
isolation and a failure to provide full and timely information. The final report of the 
Columbia Accident Investigation Board (CAIB) noted that for both the Columbia 
and Challenger accidents, “there were moments when management definitions of 
risk might have been reversed were it not for the many missing signals — an absence 
of trend analysis, imagery data not obtained, concerns not voiced, information over- 
looked or dropped from briefings.” The Chairman of the CAIB, Retired Navy Admi- 
ral Harold Gehman pointed out that NASA tends to initially follow safety proce- 
dures quite well, but then loses its diligence as time progresses. Columbia investiga- 
tion board member Air Force Major General John Barry stated that “there is still 
e'vidence of a silent safety program with echoes of Challenger.” Safety and silence 
are simply incompatible. 

The culture of aviation safety has been built on constant critical self examination, 
in an open environment, with full sharing of all the facts and analysis. Because we 
are safer today than yesterday does not mean that we cannot be safer tomorrow. 
It also doesn’t mean that our gains are not perishable. For example, on July 2, 1994 
USAir flight 1016 crashed in Charlotte, North Carolina. We determined that the 
causal factor was something we hadn’t seen in the United States in almost a decade: 
wind shear. Wind shear detection equipment and improved pilot training had all but 
eliminated this hazard and yet more sophisticated weather detection equipment — 
Terminal Doppler Radar — had fallen years behind schedule due to procurement and 
design problems. 

Furthermore, because we have made major accidents such a rarity, our ability to 
identify risks, and maintain or increase safety now depends primarily on our ability 
to fully analyze incidents and trends. In the absence of a major fatality accident or 
without a complete picture of runway incursions, wildlife strikes, and near-misses, 
we may be lulled into a false sense of security — only to have that eventually broken 
by a catastrophic loss of life. A true safety culture requires transparency and con- 
stant vigilance. 

This vigilance is required of all involved in the aviation industry, but its absence 
is perhaps most glaring when it is the fault of government, the servants of the 
American people. As Chairman of the NTSB, I followed the dictum of Benjamin 
Franklin, who said, “The man who does things makes many mistakes, but he never 
makes the biggest mistake of all — doing nothing.” I never wanted the American peo- 
ple to think that, when a need was identified — as it was in any number of safety- 
sensitive issues — we did nothing. Let us then not shrink from action but rather call 
on NASA to release its information, the denial of which flies in the face of aviation 
history, responsible government, and common sense. 

Conclusion 

We are clearly facing a new generation of risks. New technology, new planes, per- 
sonnel shortages, and a massive projected increase in air travel mean that new haz- 
ards are approaching. Before we push the panic button, however, we should remem- 
ber that we have been in this situation before. In 1996, we projected an increase 
of 220 million passengers in the next ten years and identifled a host of technological 
and operational concerns that would compound this development. In response the 
President formed a commission and its recommendations — though not perfect and 
not all implemented — contributed to a substantial reduction in fatal accidents. 
Today in 2007, we are forecasting an increase of 260 million passengers in the next 
eight years and an increase of 1.5 billion in the next twenty. We have personnel 
shortages looming or already underway and have committed ourselves to new tech- 
nology. In fact the only major difference between 1996 and 2007 was 1996’s dra- 
matic and tragic loss of 340 lives in two accidents. 
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Congress, government agencies, and the aviation industry must once again come 
together to address the rapidly changing aviation environment. We must stay ahead 
of events instead of waiting for another crash. Steps must be taken to prevent a 
deterioration of our nation’s aviation safety culture, a deterioration that NASA’s de- 
nial of transparency plainly represents. In only such a manner can we adapt to a 
growing and diversif 3 dng industry with a rigid adherence and commitment to the 
safety of all who fly in our nation’s airspace. 
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Discussion 

Chairman Gordon. Thank you, Mr. Hall. At this point we will 
open it for our first round of questions, and the Chair recognizes 
himself. 

Let me first state that I think by any measure you might take, 
particularly if you want to say the number of miles flown that 
United States has the safest air transportation system in the 
world. I fly, as Mr. Hall says, almost every week, often with my 
wife and my daughter. I don’t intend to change those flight plans 
in any way, so our discussion today is not safety and non-safety. 
It is safety and more safety. And so we should make that very 
clear. 

And let me also say that, you know, 24,000 commercial pilots and 
5,000 private pilots, I mean, that to me sounds like an unprece- 
dented amount for a survey, and so that is an enormous amount 
of data that I think should be made available, and although I rec- 
ognize NASA’s interest in a particular methodology, I think that of- 
tentimes some of the most important discoveries in America have 
been those offshoots of information. 

Release of NASA Report 

So I would ask you. Director Griffin, now that your lawyers have 
for over a year had this request on the Freedom of Information, 
when can we, why can’t this material be released today? 

Dr. Griffin. When we look at the material, despite the certifi- 
cations that you — that I know you have heard from the contractor 
involved, the data, in fact, today could not in its fullness be legally 
released. 

Chairman Gordon. And why is that? 

Dr. Griffin. Because it does contain specific comments that iden- 
tify certain airlines. It contains — it notes accidents and incidents or 
occurrences that sight specific timeframes, specific airports, specific 
makes and models of airplanes. If I look at that data, I can recon- 
struct for you 

Chairman Gordon. Dr. Griffin 

Dr. Griffin. — and so we are going to delete those fields. We are 
asking our contractor to delete those fields and to render data back 
to us which is not identifiable as they were originally required to 
do. 

Chairman GORDON. Director, I only have five minutes. I am 
sorry. 

We have asked your lawyers to cite that you were nice enough 
to provide the information to us. We couldn’t find it. We have 
asked your lawyers to point us in that direction to that informa- 
tion. They couldn’t do it. Have you seen the specific information? 

Reasons for Not Releasing Parts of the Report 

Dr. Griffin. I have seen examples of specific information which 
would not be 

Chairman Gordon. In this report? 

Dr. Griffin. In this report which would not be releasable. 

Chairman GORDON. Okay. Well, it would have been helpful if 
your lawyers had shown us, because we specifically asked that. But 
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let me also — I want to put up a slide if I could, please, from your 
contractor. Apparently there is a program that is supposed to scrub 
it, and within NASA’s own information it says, participant con- 
fidentiality is assured. So apparently you have already done this. 

Dr. Griffin. Well, no. That information is not as it stands cor- 
rect. 

Chairman Gordon. Even though it has a NASA logo on it? 

Dr. Griffin. I am sorry. It is not correct. Okay. It is possible to 
look at this data, and if one knows anything about aviation, in 
some cases to go back and identify the participants, and that can’t 
be allowed. 

Chairman GORDON. So NASA was premature in certifying its 
confidentiality? 

Dr. Griffin. Correct. 

Chairman Gordon. All right. Well, let me ask you this. You are 
familiar with the Aviation Safety Reporting System. 

Dr. Griffin. Very much so. 

Chairman Gordon. Okay. Let me just — I want to read to you one 
section of that that is from March of 2004. And this is up on the 
Internet. This is available for everybody. “After two previous,” and 
I am quoting. “After two previous red-eyes, this being the third red- 
eye in a row, the last 45 minutes of flight I fell asleep and so did 
the first officer, missed all calls from the air traffic control.” That 
was the quote. This is a report made by an aircraft crew member 
who slept through their decent, clearance, 60 miles southeast of 
Denver. Once they are awakened by the frantic calls from the air 
traffic control, they executed a successful landing. 

Now, this is just one of thousands of the reports that identify the 
airport, sometimes the approximate time, aircraft, runway num- 
bers. This material is public. 

Dr. Griffin. That is true. 

Chairman GORDON. So why should your survey not be public? Is 
it going to go into more, I mean, have you not, have they not done 
what they said they were going to do and scrub it to at least this 
extent? 

Dr. Griffin. When we look at the data, we do not at this point 
believe the data has been scrubbed sufficiently to assure confiden- 
tiality of the participants and to protect confidential commercial in- 
formation according to the standard to which we are held. As soon 
as we can do that, we will release the data. Now 

Chairman GORDON. Are you going to have a standard higher 
than this ASRS? 

Dr. Griffin. I wouldn’t say so. 

Chairman Gordon. Okay. So the information that I just read to 
you that is already public, you would not say that has to be 
scrubbed. They have to be greater, I don’t know how, you know, a 
greater level of detail to be scrubbed? 

Dr. Griffin. I don’t know that I would characterize it as a great- 
er or lesser level of detail, but we do need to remove specific ref- 
erences to airlines, specific references to incidents and timeframes 
such that pilot identity could be reconstructed. We think that that 
would be a relatively straightforward process to delete certain of 
the fields which convey that information, and we believe the initial 
release of the data could occur by the end of this year. 
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Chairman GORDON. And so you are going to do it by fields, so it 
will be by a computer program? 

Dr. Griffin. Right. Certain of the fields will be 

Chairman GORDON. Okay. Well, it seems like that is what has 
already been done here, and if it is going to computer program, 
why can’t you do it today, tomorrow. 

Dr. Griffin. I think you 

Chairman Gordon. before the end of the year? 

Dr. Griffin. — maybe, when you look at that view graph, there 
may be some confusion between anonymizing the data to satisfy 
Privacy Act considerations and rendering the data such that no one 
knowledgeable in the field of aviation could go back and recon- 
struct it. 

Chairman Gordon. Well, isn’t that the same thing? 

Dr. Griffin. I am not trying 

Chairman Gordon. If it was the Freedom of Information that the 
AP asked for this from the Freedom of Information, then wouldn’t 
you have assumed it would be made public record? And so it is the 
same thing, the same level of caution? 

And you folks had a year to do this already. 

Dr. Griffin. I don’t think we have had a year since the original 
submission, since the submission of the FOIA request. 

In any case, I am not defending, I stated for the record, and I 
will state for the record again that I believe the FOIA determina- 
tion that we should not release the data was incorrect, okay? We 
will release the data. As we set out to look at the data, to verify 
whether we could release it or not, we found that the data had not, 
in fact, been correctly scrubbed to remove identifying data. And if 
it had been, I would have released it on the spot, but it has not, 
and so until and unless I can verify that it has been correctly 
scrubbed, it will not be released. 

Information About the Data That Was Released 

Chairman Gordon. Okay. I don’t want to infringe on my time. 
You have never given me a reason not to trust your statement in 
any way. Let me just tell you that we have asked your lawyers spe- 
cifically to provide us that information, to point some place. We 
have not been, you know, you have given us data. 

Dr. Griffin. Yes, sir. 

Chairman Gordon. So all you got to do is say, look here, look 
there. And so it would give me a greater level of confidence if your 
folks could tell us where and could give us one example. Then we 
could feel more comfortable that you need this additional time. 

Dr. Griffin. Yes, sir. Let me then take that request for the 
record, and we will provide you with a couple, at least a couple of 
examples 

Chairman Gordon. Okay. 

Dr. Griffin. — where specific identifying information is included 
that would allow pilot, participant identities to be compromised. 
They do exist, and we will provide those for you. 

Chairman Gordon. And I would hope there would be to a great- 
er clarity than what is already of public record on the ASRS. 

Dr. Griffin. They are extraordinarily clear. 

Chairman Gordon. Thank you 
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Dr. Griffin. I will provide that. 

[The information follows:] 

Material for the Record 

One way, but not the only way, by which the identification of a NAOMS survey 
respondent can potentially be determined is by combining the free-text fields (pilots’ 
open-ended responses and clarifications) with data from other parts of the survey 
and/or external (exogenous) data sources. 

The availability of exogenous databases and sophisticated search technology 
makes the likelihood of implicit identification greater, and it is correspondingly 
more difficult to ensure that adequate protections have heen implemented. 

The following two examples cite free-text field responses to Question ER 1 of Sec- 
tion B (Safety Related Events), which asked pilots how many times in the past 60 
days an aircraft, on which they were a crew member, was diverted to an alternative 
airport and provide the cause for the diversion. 

Example I (Case ID 90P0001): the pilot responded, “Earthquake in Seattle.” A web 
search reveals the only seismic event that diverted flights from the Seattle-Tacoma 
Airport during the survey period: A magnitude 6.8 earthquake on Eebruary 28, 
2001. During the period of closure and reduced operations that day, approximately 
100 arriving flights were diverted; the exact number, together with airline and flight 
identifiers, could be obtained from Federal Aviation Administration (FAA), airline, 
and/or airport, databases. 

• This single response has reduced the number of candidate responders from 
over 60,000 (the number of air-carrier certificated pilots listed in the Airmen’s 
Registry) to approximately 200. 

• This profile can be further refined using non-redacted NAOMS data from 
other questions. 

- From Section A, we can determine the pilot’s flight profile for the past 
60 days (e.g., number of hours and flight legs flown; the makes, models, 
and series of aircraft flown; whether flights were passenger or cargo, 
whether the pilot flew as captain or first officer, whether the pilot flies 
for a small, medium, or large operator; and the pilot’s total commercial 
flight hours). 

- From Section B, if the pilot gave a positive response to any reportable 
safety event, an individual could cross-reference the pilot profile to event 
reports (FAA, airline) from the defined interview window (i.e., February 
28th ± 60 days) to match an individual’s name to the profile. If not, the 
profile may still match a name on airline duty rosters or other exogenous 
databases. 

Example II (Case ID 90C2001): the pilot’s stated cause for diversion was, “Amer- 
ican 587 crashed at JFK. R was en-route to JFK at the time and was diverted to 
Philadelphia.” Again, this free-text field response provides a specific event (the 
crash of an Airbus A300-600 into Belle Harbor at 9:17 AM local time on November 
12, 2001) for which there are detailed records of diverted flights. The pilot has also 
specified the alternate airport (Philadelphia), further limiting the field of possible 
flights. As before, the respondent’s profile could be refined by the non-redacted 
NAOMS data. When cross-correlated with exogenous databases, the refined profile 
might again lead to the identification of a NAOMS survey respondent. 

Chairman Gordon. — ^very much, sir. And I now recognize Mr. 
Hall. 


Confidentiality of Information About Pilots and 
Commercial Information 

Mr. Hall of Texas. Dr. Griffin, I will get right down to the basis 
of this, and we are talking about confidentiality at this time. When, 
and you can give me a yes or no answer on this I think, when pi- 
lots were surveyed, were they led to believe that their responses 
would be confidential? 

Dr. Griffin. They were promised confidentiality. Yes. 
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Mr. Hall of Texas. So if their responses were released, do you 
think it would have had a chilling affect on their future participa- 
tion in FAA or NASA surveys, and would airline safety ultimately 
he hurt by disclosing this data if fewer pilots contributed to other 
surveys and reporting systems? 

Or let me go on a little bit — ^be a little more personal with that. 
As a pilot yourself with many years of flying experience, would this 
data give you pause as to whether it is really anonymous, or would 
it worry you that your input could be traced back to you? And 
would that have a chilling affect on you? 

Dr. Griffin. Well, in its present form some of the examples can 
be traced back to pilots and some named individual airlines. That 
can’t be allowed. If the data is properly rendered untraceable, then 
I think it must be released and should be released and will be re- 
leased as I have stated several times. 

So if it were properly anonymized, I have no concern. 

Mr. Hall of Texas. And they do that by cross referencing flight 
routes, times, and carriers? 

Dr. Griffin. We need to delete the fields that contain that infor- 
mation. So that will be done. Now, the major concern I would have 
over this data at this point is that somebody might put too much 
credence in it. It is simply not credible to believe that the aviation 
community is experiencing nearly four times the number of engine 
failures that are being documented by the FAA. That is not cred- 
ible to believe. If it is true, it is going to require some very strong 
justification, and we will pursue that. The community will pursue 
that, but it is not credible at this point. 

So I would not want the flying public to believe the data in the 
form that it appears today. 

Mr. Hall of Texas. Mr. Hall, wake up. 

Mr. Hall. Yes, sir. 

Mr. Hall of Texas. NASA surveys but one source of data that 
can be used to major safety transit and National Airspace System. 
What other sources can be used to monitor system safety, and how 
useful are they? 

Mr. Hall. Well, the ASRS System, which NASA has used for 
years, is, of course, I think very useful in terms of it is a voluntary 
program. It is a program that — that is why I am a little confused 
on — in regard to some of the comments from the Administrator. 
NASA has run this program for the Federal Government for a 
number of years. So they are familiar with how to put a program 
together and maintain confidentiality. 

There are other programs that run by FAA and, of course, NTSB 
has gotten into trying to look at as many incidents as possible in 
providing information. But aviation safety benefits from having, as 
I mentioned in my statement, sir, a very open system and a system 
where there is a whole lot of information and that information is 
constantly in the public for analysis and review. 

Mr. Hall of Texas. Dr. Griffin, you state that NASA will release 
this survey data so long as it doesn’t compromise the anonymity of 
the pilots, keep them anonymous. Does that not contain confiden- 
tial commercial information? 

Dr. Griffin. Well, by the time we release it, it will not contain 
confidential commercial information. Some of the data that we 
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have today does, and we are not legally allowed to do that by stat- 
ute. 

You know, there have been a number of comparisons made to the 
Aviation System Reporting System, the ASRS, which NASA does 
manage by statute, and this survey. One of the primary differences 
between ASRS and this survey was that ASRS is managed by avia- 
tion specialists. When reports are made, the aviation specialists 
can contact the submitter of the report and ask follow-up questions. 
They are knowledgeable about aviation safety. 

This survey was conducted by telephone polling surveyors, who 
have no knowledge or had no knowledge at all as to aviation or 
aviation safety. They had no domain expertise, and it is precisely 
that which has led to some of the problems that we are here dis- 
cussing today. 

Mr. Hall of Texas. I think my time is up, Mr. Chairman. I 
thank you. I yield back if I have any. 

Chairman GORDON. Thank you, Mr. Hall. Let me once again 
state that this is not a matter of a hearing between a safe system 
and an unsafe system. It is a matter of a very safe system that we 
want to, you know, make, continuing the model for the entire 
world. 

Now we will recognize Mr. Costello, the Chairman of the Avia- 
tion Subcommittee of the Transportation and Infrastructure Com- 
mittee. 

Mr. Costello. Thank you, Mr. Chairman, and thank you for 
calling this hearing today as well as the Chairman of the Sub- 
committee, Chairman Miller. Welcome, Dr. Griffin, Mr. Hall. Mr. 
Hall, it is good to see you in Science Committee room for a change 
as opposed to the T and I room. 

But let me — we can go through a whole long list of questions. Let 
us cut to the chase and get down to why we are here. 

Getting the Information to the Public 

We talk about scrubbing the report in order for it to be released 

without breaking anyone’s confidence or a commit . You are 

saying that you can release the information possibly by the end of 
the year. Is that correct? 

Dr. Grifein. Yes, sir. 

Mr. Costello. How long will it take, I mean, if it is a priority 
in the agency, we all, and I would hope that you would acknowl- 
edge that the agency made a huge mistake in how they responded 
to the AP and to the media. Your spokesperson did, in fact, unless 
you are refuting this, did, in fact, say to the news media that, if 
we release the data it could be, could have an adverse affect on the 
industry. Is that correct. Dr. Griffin? 

Dr. Griffin. We did say that, and as I have now said several 
times, that was the wrong thing to have said. I apologize that any- 
one in my agency did say that. 

Mr. Costello. So you know that it was a mistake to say that. 
You know that it has created a lot of controversy. You know that 
people in the aviation industry and the traveling public, because I 
have heard from my constituents, and I have heard from complete 
strangers to me at airports as I am flying, what is going on with 
this report, and what won’t you release it to the public? 
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If it is a priority to us, shouldn’t it be a priority to your agency 
to scrub this and get it out to the public immediately? 

Dr. Griffin. It is a priority. I have spent, I have a Shuttle mis- 
sion in the air right now, and I have spent little else this past week 
except to work on this issue. I regret 

Mr. Costello. I would hope that there are other people in the 
agency that you could assign this to as opposed to you handling 
this personally. 

Dr. Griffin. Well, we have had quite a number of people work- 
ing on it. We do consider it to be a priority, and we consider it to 
be an important one. Now, the fact that people at NASA misspoke 
concerning the reasons behind the denial of the FOIA request does 
not mean that we can compromise our statutory requirements 

Mr. Costello. And no one is asking you 

Dr. Griffin. — on FOIA. 

Mr. Costello. — compromise a statutory requirement. 

Dr. Griffin. Right. 

Mr. Costello. What we are saying is get this done and get it 
out to the public, and my question to you is do you have people 
today and this evening and around the clock working on this 
project to scrub it to get it out to the public? 

Dr. Griffin. The people who have to work on this project to 
scrub the data and get it to the — out to the public are at Battelle 
Institute. They have been directed to do that. I hope that they are 
doing that with all deliberate speed, and we will be verifying that. 
When Battelle has finished scrubbing it, the quality of the scrub 
must be judged by government officials, who will then do that as 
quickly as possible, and we will get it out to you. 

Mr. Costello. So have you directed Battelle to work on this 
around the clock? Have you given them a deadline? 

Dr. Griffin. I have not directed them to work on it around the 
clock. We have directed them to work on it. 

Mr. Costello. Isn’t it reasonable for us to expect for you to give 
them a deadline? They are working for you. 

Dr. Griffin. They are, and we have asked them to complete it 
by the end of the year. That is what we are asking. That is two 
months away. 

Mr. Costello. And if you told them June of ’08, they would com- 
plete it in June of ’08. Isn’t that correct? 

Dr. Griffin. You are asking for more detail than I have. It is a 
significant amount of data processing. We will do it as soon as we 
can, and we are trying for the end of the year. 

Mr. Costello. Dr. Griffin, you have acknowledged that the agen- 
cy misspoke. They created this uproar with the American people 
and with the Congress and with everyone in this room. It is your 
responsibility to clean this up. 

Dr. Griffin. That is correct. 

Mr. Costello. If I were in your shoes, I would be directing 
Battelle to work 24 hours a day, seven days a week to get this 
thing cleaned up so it can be released to the public. 

Disciplinary Action for Responsible Party 

Last and final question that I have, the person who misspoke 
representing the agency, have you identified who that is? 



32 


Dr. Griffin. Yes, sir. 

Mr. Costello. Have you taken any disciplinary action against 
that person? 

Dr. Griffin. It is not a matter of discipline. People make mis- 
takes. This was a mistake. 

Mr. Costello. My question, we all understand it was a mistake. 
Has there been any disciplinary action taken? 

Dr. Griffin. No. 

Mr. Costello. Thank you, Mr. Chairman. 

Chairman Gordon. The gentleman from Wisconsin, former 
Chairman of the Aviation Subcommittee on this committee, as well 
as the Full Committee, Mr. Sensenbrenner, is recognized. 

Mr. Sensenbrenner. Thank you very much. 

NASA Survey and Confidentiality 

Dr. Griffin, first of all, I think we all want to see what the re- 
sults of the survey are. Secondly, I think we all agree that certain 
things have to be kept confidential. That was what was rep- 
resented to the people who were asked to respond to the survey, 
and they responded candidly based upon the representation of con- 
fidentiality. 

I guess what I would like to know is the survey was finished in 
2005, and we are almost at the end of 2007. That is two and one- 
half years more or less between the time the survey was finished. 
Why is there this gap in time? Who dropped the ball? 

Dr. Griffin. We at NASA did not manage this project to its con- 
clusion well. We did not. Because of that I have instituted a look 
at other projects that we are doing in various classes of research 
at NASA to make sure that we are not doing the same thing else- 
where. 

Mr. Sensenbrenner. Which NASA center of “excellence” super- 
vised Battelle and this survey? 

Dr. Griffin. This particular project was supervised out of the 
Ames Research Center. 

[The information follows:] 

Material for the Record 

NASA Ames Research Center agrees that a more timely report on NAOMS should 
have been provided. The NAOMS contractor team consisted of a small group of indi- 
viduals who supported a few related projects. The NASA NAOMS project manage- 
ment officials decided to allow the contractor team to defer preparing a timely re- 
port in order to conduct other activities in support of the NAOMS project, notably 
the transition of the NAOMS survey methodology, as well as to address priorities 
in other projects they were supporting. In the process, attention was diverted from 
the final report, resulting in an inordinately lengthy delay. 

The NAOMS contractor completed the survey collection in December 2004. In FY 

2005, the NASA NAOMS project management officials prioritized project resources 
to enable the transfer of the NAOMS methodology to a new host organization. This 
transfer required adapting the NAOMS data collection methodology from a com- 
puter-aided telephone interview to a web-based format. Throughout FY 2005 and FY 

2006, the NAOMS contractor team was thus directed to develop the new method- 
ology, in collaboration with NASA researchers, and to transfer the methodology to 
the Air Line Pilots Association (ALFA), under the auspices of the Joint Implementa- 
tion Measurement and Data Team (JIMDAT, the evaluation arm of the Commercial 
Aviation Safety Team). 

By early FY 2007, the NAOMS project team had not completed the transition of 
the methodology to ALFA nor had the contractor completed its final report. By this 
time, the contractor was needed to support the Aviation Safety Program priority to 
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develop safety data mining tools. The NASA NAOMS project management officials, 
therefore, directed the contractor to focus on this priority and provided an extension 
to the contractor for producing a final report on NAOMS. Proper attention is now 
being given to producing this report, and measures will be taken to ensure that this 
kind of delay on contract deliverables does not happen in the future. 

Mr. Sensenbrenner. Okay. Have you found out why the Ames 
Research Center didn’t follow up and have a timely report? 

Dr. Griffin. I have not. 

Mr. Sensenbrenner. Will you do it and let us know? 

Dr. Griffin. I will take that for the record. We will find out what 
their rationale was for taking so long to allow this report to he gen- 
erated, and we will answer back to you. 

Mr. Sensenbrenner. Okay. Well, let me say that this appears 
to be a mess of NASA’s own causing, and you are the agency head, 
and I would hope that we don’t hear from you again on another 
mess of NASA’s own causing. 

You know, I would point out that in about two and a half years 
we are going to have a census in this country, and one of the things 
the Census Bureau represents to every American or everybody who 
is in this country, is that their responses will be confidential. And 
that is in order to get a candid response on not only how many peo- 
ple are here but the housing questions and the other things that 
are asked on the census form. 

Any government agency that gets itself caught in a pickle like 
NASA is in is going to reduce the confidence of the American public 
that responses that are supposed to be kept confidential will indeed 
be kept confidential. Sir, you dug yourself into a hole. I can’t say 
that you are not digging yourself deeper into the hole from what 
I have heard at this hearing, but I think it is important more than 
just for your agency but the government as a whole that you start 
working yourself out of that hole. 

Thank you, and I yield back the balance of my time. 

Chairman Gordon. The gentleman from Colorado, the Chairman 
of the Space and Aeronautics Subcommittee, Mr. Udall is recog- 
nized. 

Mr. Udall. Thank you, Mr. Chairman. Welcome, Dr. Griffin. 

Releasing Information and Why Was the Survey 
Ended? 

I would like to start by echoing what Chairman Gordon said 
today. We are all disappointed we have had to convene the hearing, 
but the fact that NASA refused to release the taxpayer-funded 
aviation safety survey and the rationale that NASA gave for refus- 
ing to release this information is unacceptable, and it obviously re- 
quired Congressional scrutiny. I think we all agree the safety of the 
public has to be our first priority, especially with more and more 
Americans flying every year. 

I am glad that you have now agreed to release at least some of 
the survey data publicly so that it can be used to help maintain 
and hopefully improve the safety of the Nation’s airways, but I feel 
strongly that all of the data should be made publicly available as 
soon as possible. 

I also have some concerns about why the study was ended. Sev- 
eral witnesses here today have affirmed the value of a comprehen- 
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sive, ongoing survey and analysis approach to aviation safety trend 
analysis and accident precursor identification, which is the ap- 
proach exemplified by the NAOMS Project. I think there appears 
that we would all agree to be a great deal of merit to the NAOMS 
approach, and we need to assess whether NASA and the FAA 
should reinstitute the project. 

Doctor, if I could just leave aside for a moment the issue of peer 
review, survey methodologies, which our second panel will be ad- 
dressing, I have to say that I am troubled by your testimony on the 
NAOMS project. At one point in the testimony you state that the 
project was not shut down prematurely and that the transition of 
the survey methodology to industry, government decision-makers 
was successfully completed. 

However, later in your testimony you say that any product of the 
NAOMS Project including the survey methodology should not be 
viewed or considered at this stage as having been validated. Basi- 
cally, at least to this Member, you are saying that NASA didn’t 
complete a critically-important R&D task, the validation of the sur- 
vey methodology before it transitioned NAOMS out of NASA. 

Later Captain McVenes will testify that the Aviation Committee 
had plans to work with NASA to help determine if the survey data 
were reliable, but funding for NAOMS ran out, and that is when 
the A-L-P-A, ALP A, stepped in to help keep the project alive. 

This doesn’t appear to be the normal way R&D programs should 
be run, and I think that the Space and Aeronautics Subcommittee 
will need to take a closer look at NASA’s aeronautics programs and 
its aviation safety programs in particular in the coming months. 
But in the spirit of openness and dialogue here, I would see if you 
care to respond to those comments. 

Dr. Griffin. Well, we certainly agree — could not agree more that 
the aviation safety information leading to trending analysis and ac- 
cident factor identification before the fact is crucial. We are work- 
ing on exactly those things in concert with the FAA, again, in a 
program that has been reviewed by the FAA’s own safety sub- 
committee, in which we have submitted for review to the National 
Academy. So we agree with that. 

NASA, however, is not the entity responsible or even allowed to 
take on the job of operational aviation safety. We do research, and 
we are doing that. And we expect to continue to do it, because we 
do believe it is important. 

Now, as I said in my testimony earlier, the National Academy in 
its 2004, review specifically stated that they did not see a reason 
for the NAOMS Project to continue. We agree. We have 
transitioned our other projects of that type to a joint FAA, NASA 
arrangement that I think is working well, and when NAOMS was, 
as it was, scheduled to end in 2004, with follow-up reporting to be 
done in 2005, we allowed that to occur as had been planned. 

So I don’t think there are — I don’t think there is any evil intent 
there. There was no intent to abrogate our responsibilities. In fact, 
our intent was to execute them as best we could with our FAA 
partner. What was not done here was to bring the project to a 
timely conclusion, to assess the data, to issue a report, to publish 
that report in peer review journals, and to release the data to the 
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public in a timely way, properly anonymized. That was not done, 
and we are going to have to do it. 

Mr. Udall. The spirit in which I offer my remarks are as follows. 
I think this situation, of course, is one that we have great concern 
about on the Committee, but I think we should take advantage of 
the clear opportunity here to make our system safer and to take 
this data, 24,000 responses, that is very, very significant, and apply 
it and use it in a way that has some utility in the coming months 
and the coming years. 

I see my time has expired. Thank you for being here today again. 

Chairman GORDON. Dr. Ehlers, thanks for your being prompt 
today. I am sorry that I overlooked that earlier and Dr. Ehlers of 
Michigan is recognized for five minutes. 

Mr. Ehlers. That is quite all right. I am used to being over- 
looked. I hope you all feel sorry for me. 

Airline Safety Compared to Other Safety Concerns 

Actually, I am going to take a somewhat different attack and 
also do some criticism but not of you. Dr. Griffin. 

Your situation reminds me very much of a quote from Harry Tru- 
man when he left the Presidency. His comment was, “This job was 
not so great. I spent all of my time trying to persuade people to 
do things they should have had sense enough to do in the first 
place.” Your situation reminds me a bit of that, and I agree with 
the comment made that you have more important things to do than 
to deal with this particular problem, and it is unfortunate that it 
developed and entwined you in it. 

But as the son of a preacher, I have to give a little sermon here, 
and I have been warned never to insult the media, but I am going 
to anyway. Because it has always puzzled me why the media are 
so obsessed with aviation safety when it is the safest mode of 
transportation in this country. I remember some years ago when I 
was new in the Congress but there was a low-cost airline that had 
an airplane crash in the Everglades because some attendant or 
some mechanic had loaded some oxygen units on the plane which 
shouldn’t have been there. Day after day, month after month this 
was headlines in the newspapers, and I pointed out repeatedly that 
the same day that airplane crashed, more people were killed in 
automobile accidents in this country than were killed in that air- 
plane. Every day after that more people were killed on the high- 
ways than were killed in that plane crash. Yet headlines day after 
day. 

The safety is better than any other mode of transportation. We 
should recognize that and participate in it. I don’t fault you what- 
soever for things that may have gone wrong in this. You were 
caught in an unfortunate situation in responding to a FOIA re- 
quest, which is a no no. But nevertheless, I think your motives 
here were very good. 

I would also point out that if people are so concerned about safe- 
ty, there is an immediate problem you can tackle with traffic acci- 
dents, and that is drunk drivers. We have had a number of drunk 
drivers kill individuals while we are sitting here in this session, 
more than were killed by airplanes. And it goes on year after year. 
In fact, so far — or — in any given year more individuals are killed 
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by drunk drivers than were killed among our troops in the entire 
Iraq War up to this day. That is every year that happens, and yet 
we spend all this time on aviation safety. 

I don’t, I am not opposed to making airlines, airplanes as safe 
as they should be. I am a would-be pilot myself, and I certainly 
want a safe airplane and safe air traffic control system. But let us 
get over this obsession and let us recognize that our goal is to im- 
prove what is already very good and not get obsessed about little 
incidents that occur when we have much bigger problems to try to 
tackle in the aviation sector. 

So I beg your apology for the sermon, Mr. Chairman, but I just 
have to say these things once in awhile. Let us get stuff in perspec- 
tive, and the world is not going to rise or fall, and the aviation in- 
dustry is not going to rise or fall on the results of this survey. I 
doubt if we will learn much different than we have learned from 
the previous surveys. It is all good. Let us all do it, but let us not 
overstate it. 

Thank you very much. I yield back. 

Chairman GORDON. Thank you. Dr. Ehlers. I hope you feel bet- 
ter. 

The gentleman from Louisiana, Mr. Melancon, is recognized for 
five minutes. Melancon passes and let me see, Mr. Mitchell from 
Arizona, also on the Transportation Committee, is recognized. 

Mr. Mitchell. Thank you very much, Mr. Chairman. This is for 
Dr. Griffin. 

You know, airline business in my particular district is a very, 
very serious business. One of the Nation’s largest airlines is located 
in Tempe and Phoenix Sky Harbor is the eighth largest or busiest 
airport in the country. We depend on aviation, and we depend upon 
the Federal Government to keep our skies safe. 

Responsibility for Public Statement 

Now, I was stunned as most people were, and I think this is why 
we are here, because of the statement that came out that you have 
heard many times before — the affect of public — the reason the re- 
port was not released is because of the affect that it might have 
on the public confidence and so on. 

Now you are telling us that you don’t agree with that statement 
that was made last week. But, Dr. Griffin, you are the Adminis- 
trator of NASA. How could this statement be released without first 
being reviewed and agreed upon as NASA’s stance on this par- 
ticular issue? 

Dr. Griffin. The delegated FOIA official released the response in 
the form of a letter and included a statement that I believed to 
have been mistaken. I try to review everything that I believe will 
be significant before it goes out, but I don’t have enough hours in 
the day to review every single thing that goes out of NASA, and 
sometimes mistakes are made. This was one, and when that occurs, 
as the agency had, I pay the price for it. 

Mr. Mitchell. But you have the time now to come and 

Dr. Griffin. Obviously I have had to make the time, because we 
did make a mistake, and the mistake rests on my shoulders, and 
I apologize for it, and I have before, and I will again. The language 
that was used was inappropriate. We will not repeat it. We will 



37 


correct the error. We will de-identify the data, and we will release 
it. 

Mr. Mitchell. Well, besides this particular statement I would 
hope that you would have a better review of what comes out of 
your office, because you may be back here again the way things 
seem, doing the same thing you are doing now. 

Why Wasn’t NASA Information Made Public and Why 
Didn’t It Live Up to NASA’s Standards? 

You know, you said that NASA is interested in getting this safe- 
ty information out, and my question is why has NASA refused to 
produce it to the Associated Press for a year? Now, my under- 
standing is the study started in April of 2001, ended in December, 
2004. Why does it take a hearing in Congress and public pressure 
for a hearing to get the public made — to get this information made 
public? 

Dr. Griffin. As I said earlier, the only way that I can answer 
that question is to admit, as I have, that we did not manage that 
project well. We did not bring it to a timely conclusion. We did not 
publish the data and the report’s conclusions in an appropriate 
way, and we will fix it, and we will try not to do it again. 

Mr. Mitchell. The next part of this question is you stated that 
this was not conducted — this survey under proper standards of 
NASA. So it seems like there has been a lot of mistakes here. And 
this is one of them you say it wasn’t under NASA’s normal review. 

Why would NASA invest over $11 million in a project like this 
if it didn’t follow NASA standards? 

Dr. Griffin. We did not manage the project well. We did not su- 
pervise our contractor appropriately. We made a mistake. 

Mr. Mitchell. You know, all of this reflects on NASA’s credi- 
bility. 

Dr. Griffin. Yes, sir, I do. 

Mr. Mitchell. I yield back. 

Dr. Griffin. I deeply regret the situation, and I will look, and 
we are now looking to make sure that this does not occur again. 

Chairman Gordon. Thank you. We are going to be having votes 
in about 20 minutes, so I am going to, I want everybody to have 
their say. I will be stricter than usual on the five minutes, and if 
you want to be briefer than usual, then that would be good, too. 

So, Mr. Bonner from Alabama is recognized. 

State of Current Space Shuttle Mission 

Mr. Bonner. Thank you, Mr. Chairman, and I could probably 
spend the next five minutes trying to think of some creative way 
to ask the same question that has been asked repeatedly to get a 
different answer, but instead, if I might, I would like to ask, take 
advantage of this opportunity that we don’t often have to ask Dr. 
Griffin how the Shuttle mission is going. Because I think a lot of 
people are interested. We have followed that with great interest 
over the years, and I think it would be great to hear from you on 
how it is going at this point. 

Dr. Griffin. It is going extremely well. We have an unfortunate 
rip in one of the solar arrays, not a huge rip, but a rip, and that 
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is important to repair before the crew returns. And so we are going 
to extend the mission an extra couple of days to do that. But other 
than that it is going extremely well. 

Mr. Bonner. Do you feel personally responsible for that solar 
rip? 

Dr. Griffin. You know, I am an ex-program manager, and my 
belief is if lightening strikes your payload, it is your fault. So, yes, 
I feel responsible for that rip, and we are, and for repairing it, and 
we are going to fix it. 

Mr. Bonner. Thank you very much. Thank you, Mr. Chairman. 

Chairman CORDON. Thank you, Mr. Bonner, and the Chairman 
of our Oversight Committee, Mr. Miller, is recognized. 

Quality of Data 

Mr. Miller. Thank you, Mr. Chairman. Dr. Griffin, good after- 
noon. The next panel includes Dr. Robert Dodd, who was a prin- 
ciple investigator for the NAOMS Project. Have you discussed your 
testimony with Dr. Dodd at all? Have you reviewed his testimony? 

Dr. Griffin. No, I have not 

Mr. Miller. Okay. 

Dr. Griffin. — met your next two witnesses. 

Mr. Miller. His prepared testimony says that the NAOMS Team 
made an extraordinary effort to clean and validate the data col- 
lected through the survey. The resulting data is of good quality and 
ready for meaningful analysis. You disagree with Dr. Dodd? 

Dr. Griffin. I do disagree with that statement. 

Mr. Miller. Okay. 

Dr. Griffin. The self-assertion by the purveyors of the data that 
the data is okay does not make it okay. 

Mr. Miller. Okay. Well, that was another, I mean, I understand 
a concern for methodology, but there does need to be an extraor- 
dinary concern for methodology. Dr. Dodd’s statement of the pur- 
pose of NAOMS was help identify risks that could result in losses, 
evaluate the impact of new technology, provide insights into how 
well the safety enhancements are working out. In other words, pro- 
vide results based upon which we could act. 

And your testimony is that the overarching goal of developing, 
was developing methods to facilitate a data-driven approach to 
aviation system safety analysis, that in early 2005, you determined 
that the amount of data collected was sufficient to evaluate, wheth- 
er NAOMS survey methodology was statistically useful. There were 
29,000 survey results. I would hope that that would be enough in 
representative. And then you said — you have said in your testi- 
mony that it was not prematurely ended. It sounds from your testi- 
mony like the purpose of the project was to develop a methodology. 

It seems like $11.3 million is a lot for methodology. That ought 
to buy you a lot of methodology. Was it your purpose to do the 
things that Dr. Dodd said, which is have information that you 
could use? 

Dr. Griffin. Well, from NASA’s perspective the purpose was to 
develop and validate methodologies and then to transition the work 
to the agencies with operational responsibility. 

Mr. Miller. Okay. And when did that transfer happen? 
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Dr. Griffin. The transfer of methodology and data to the Air 
Line Pilots Association, which had expressed some interest in a 
weh-based version of the survey, occurred in 2004, 2005, and 2006. 
NASA has briefed the results of the study to the FAA, among other 
government agencies. 

Mr. Miller. So it has been analyzed to that extent? 

Dr. Griffin. It has been analyzed to that extent, and that anal- 
ysis that you refer to that has been done to that extent revealed 
substantial concerns. For example, if you were — if you extrapolate 
the rate of certain things done, revealed by the survey, you get an 
uncredible answer. 

For example, pilots were asked how often they had to land an 
airplane at an unscheduled airport in order to deal with an unruly 
passenger. We accumulated those statistics. If those statistics are 
extrapolated forward, it yields a result that four times a day a 
transport aircraft is landing because the crew has to deal with an 
unruly passenger. 

Now, I recall since 9-11 that that has happened maybe two or 
three times. 

Mr. Miller. Okay. 

Dr. Griffin. If we had people landing four times per day to deal 
with an unruly passenger, it would be on the nightly news every 
night. That is not happening. So it causes us to suspect the quality 
of this data. 

Mr. Miller. All right. Dr. Griffin, I understand that the Office 
of Management Budget has an office of experts, of survey experts, 
survey methodology is not unusual. It is widely used in the Federal 
Government. It is widely used in social sciences. It is widely used. 
Survey experts and statisticians who review the methodology of all 
surveys used by the Federal Government. Was this survey re- 
viewed by that office of 0MB? 

Dr. Griffin. I don’t know. I was not at NASA when that work 
was done, and so I don’t know if it was reviewed by the 0MB at 
that time or not. 

Mr. Miller. All right. You cite as still correct the refusal to pro- 
vide the information under FOIA as revealing confidential commer- 
cial information. My understanding of that exception is that that 
is to protect the confidentiality of information provided by a busi- 
ness entity that might be confidential for business reasons. Market 
information, financial information, et cetera. 

It is hard to see how this survey data provided by pilots would 
meet that exception. What kinds of confidential commercial infor- 
mation did this survey produce? 

Dr. Griffin. Well, the exemption that you refer to is, of course, 
correct as you state, but it is not the only one. In the case where 
information is voluntarily provided and when that information 
would not be customarily provided to the public, then we also have 
an obligation to protect that information. 

Chairman Gordon. The gentleman’s time has expired. 

Mr. Inglis is recognized. 

Mr. Inglis. I pass, Mr. Chairman. 

Chairman GORDON. The gentleman, Mr. — thank you. And is 
there — Mr. Lipinski. Excuse me. Mr. Chandler is next, then Mr. Li- 
pinski and 
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The Responsibility for the $11 Million 

Mr. Chandler. Dr. Griffin, I have been listening to the testi- 
mony, and I understand that, it sounds to me like you may believe 
you all made a mistake. 

Dr. Griffin. I have admitted it several times. 

Mr. Chandler. I think that has come out in this hearing. And 
I understand that has to do with the handling of the FOIA request. 

Dr. Griffin. Yes, sir. 

Mr. Chandler. But I also just — it just came across my mind that 
maybe you believe that this entire process has been mishandled, 
and you have made a mistake in the entire survey process and not 
overseeing what is a pretty enormous project. Is that the case as 
well? 

Dr. Griffin. I have — this is not an enormous project by NASA’s 
standards. 

Mr. Chandler. No, but in this particular instance it is a pretty 
important project. 

Dr. Griffin. But when we spend $11 million of the taxpayers’ 
money it should be done well, and I have stated — I regret to state 
it, but I have stated that by my standards we did not manage this 
project well. We did not manage our contractor well. 

Mr. Chandler. And you are also saying that at the end of all of 
this and when this data is, in fact, released, there is going to be 
reason to not have much confidence in the ultimate data. Is that 
correct? 

Dr. Griffin. I have been a pilot for decades. Anyone who knows 
anything about aviation is going to look at this data and have a 
lot of questions about it because it is on its face — on its face, when 
you look at it, you can extract from it conclusions which are not 
credible. 

Mr. Chandler. Well, what I am hearing you say is we have just 
thrown $11 million down a rat hole. 

Dr. Griffin. I hope that is not the case, and I believe that we 
should be able to get much that is useful from this data, but there 
will be cause to question it by knowledgeable aviation experts. 

Chairman Gordon. Would my friend yield to Mr. Lipinski so we 
can try to finish this panel? 

Mr. Chandler. Sure. Thank you, Mr. Chairman. 

Chairman Gordon. Thank you. 

Mr. Lipinski. I will try to make this quick, although this is very 
important. Airline safety is critical. I have an airport in my dis- 
trict, O’Hara Airport very close proximity. I just want to zero in. 
Dr. Griffin, I have a lot of respect for you. Today you are on the 
hot seat, deservedly so with this project. 

You talk about this project was not managed well. To me I look 
and see the project started six years, $11 million, no results. It 
could mean one of two things. Either complete incompetence, that 
this project had so many problems with it, that you couldn’t get 
anything good out of it, or, you know, I could use the word cover 
up, I will say, but there, or there is some reason that this was 
stopped. There was something that, for some reason someone did 
not want to show up. 
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When this stopped, were there plans for anyone else to be sur- 
veyed after you did the airline pilots? Was there anyone else after 
that? 

Dr. Griffin. No. There was not. This project 

Mr. Lipinski. Was that the end? 

Dr. Griffin. — in the original material, which has been submitted 
to this committee, documenting this project, it was intended that 
the project be ended in 2004. We have for purposes of transition 
and simply because things have gone slower than they should have, 
this project has continued onto the present day. But there has been 
no cover up. There is no desire to conceal anything. 

Mr. Lipinski. Okay. I am very 

Chairman GORDON. Mr. Lipinski, would you mind yielding to 
Ms. 

Mr. Lipinski. Yes, I will yield. 

Chairman GORDON. — Richardson to, for her concluding state- 
ment? 

Ms. Richardson. Thank you, Mr. Chairman. I will just be very 
brief or as quick as I possibly can. 

Dr. Griffin, I represent the California area, and we have had sev- 
eral reported incidences within the LAX Airport, and I also rep- 
resent the Long Beach Airport. I have the following questions, and 
if you can’t answer them within the time we have provided, you 
can provide them to this committee. 

Data Recovery, Peer Review, and Avoidance of 
Requests 

Number one, who and when decided that there would be a de- 
struction of data requested? In your statement you say that that 
didn’t happen, and so my question to you would be if it didn’t hap- 
pen, then why was it requested that the subcontractor — why were 
they directed to recover data? It just doesn’t make sense. If they 
weren’t required to destruct it, then they should now be required 
to recover it. 

Chairman GORDON. If the gentle lady, would you go ahead and 
read your questions and then Dr. Griffin can respond for the record 
if that is okay. 

Dr. Griffin. We will take them for the record. Yeah. 

Ms. Richardson. The second thing is if the project was initiated 
in 1998, started collection in April of 2001, and started that in 
2004, I find it really hard to understand, number one, why in 
seven, eight years you failed to complete a peer review, why we 
now suddenly question the methodology. I come from the private 
sector. I don’t know of anyone who manages a project that you 
don’t look at the data, how the data is being collected, how is it 
being presented, how are you going to use it, what should be in- 
cluded, what should not be included. That we finally wake up eight 
years later? I have never — I don’t know of a system of how we do 
this and we operate it. 

And then finally, I would say really the continued avoidance of 
requests is just unprofessional. I am a new Member here, but I will 
tell you what I call it. I don’t call it a mistake. I call it negligence, 
and I really think that NASA is liable, and if something happens, 
this is a very serious issue, and I really resent that we are here 
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today even having this discussion. This is something that could 
have been dealt with, I believe, if you really wanted it to. And for 
me to say two months is completely unacceptable. These are com- 
puter programs, you either make it a priority or you don’t, and it 
seems to me today it is not a priority to you. 

Thank you. 

[The information follows:] 

Material for the Record 

NASA Ames Contracting Officer issued the phase-out instruction to Battelle Inc. 
on September 10, 2007, via a new task request; this task instruction was made in 
preparation for task phase-out scheduled for October 31, 2007. Per this task instruc- 
tion, written in order to properly disposition sensitive government information, the 
Ames Contracting Officer instructed Battelle Inc. to collect, inventory, archive, and 
transfer the complete set of data to the government. Once Battelle Inc. completed 
this transfer, and the NASA project management officials verified the completeness 
of the data set, Battelle Inc. was instructed to securely dispose of all data. This in- 
struction was to ensure that the data set was NASA-owned and to prevent the po- 
tential for unauthorized use of the data. 

NASA received a letter, dated October 22, 2007, jointly signed by Chairman Gor- 
don, House Science and Technology Committee, Chairman Udall, House Space and 
Aeronautics Subcommittee, and Chairman Miller, House Investigations and Over- 
sight Subcommittee, directing that NASA halt any destruction of records related to 
the NAOMS project. To comply with the direction, the Ames Contracting Officer di- 
rected the contractor to halt the phase-out process until further notice. This action 
was done via a task modification dated November 5, 2007. 

NASA Ames Research Center agrees that the methodology should have been peer- 
reviewed much earlier in its development. While the survey was approved by the 
0MB in accordance with the Paperwork Reduction Act, and briefed to stakeholders 
in two workshops, the work was not peer-reviewed. 

From 1998 to 2004, the NAOMS project team gave approximately 17 Power Point 
briefings to various audiences, mainly government and industry personnel. How- 
ever, none of the research conducted in the NAOMS project has been peer-reviewed 
to date. Power Point briefings to stakeholders, while having some value, do not con- 
stitute peer review. Accordingly, no product of the NAOMS project, including the 
survey methodology, the survey data, and any analysis of those data, should be 
viewed or considered at this stage as having been validated. 

It should be noted that NASA’s assertion that none of results from the NAOMS 
project can be considered validated does not mean that NASA is drawing conclu- 
sions about the validity of the survey data; we are simply stating that no such con- 
clusions can be credibly drawn. 

In order to rectify this situation as best as possible, NASA has asked the National 
Academies to conduct an independent assessment of the contractor’s final report as 
well as of the survey results that are to be publicly released. The National Acad- 
emies’ assessment will be made available to the public as soon as it is completed. 

Mr. Hall of Texas. Mr. Chairman. 

Chairman Gordon. Yes, Mr. Hall. 

Mr. Hall of Texas. Could I make an inquiry of 

Chairman Gordon. Certainly. 

Mr. Hall of Texas. Mike, would you mind staying around dur- 
ing the second panel where we might respond to anything else that 
might happen? You know, something may come up as to whether 
or not we have handled immigration well, you know, the whole 
Congress might get indicted on that. We may have some questions 
on why we don’t have an appropriations bill for the first time in 
history. A lot of us haven’t handled things well, and you have said 
that you haven’t, you acknowledged it. Please stay around, if you 
would, for this next — to where we can inquire of you for some an- 
swers if we need them. Would you? 

Dr. Griffin. Of course. Yes, sir. 

Mr. Hall of Texas. Thank you. 
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Chairman GORDON. Mr. Hall of Tennessee, thank you for being 
here. Dr. Griffin, you are a good Administrator of NASA, the buck 
stops with you. It is unfortunate you have to spend this time. I 
hope the message goes out to those folks that work for you that 
they should not put you in this position in the future. 

We will take a recess to go vote and then come back for our sec- 
ond panel shortly. 

[Recess.] 

Chairman GORDON. As a courtesy to our witnesses if everyone 
would come back and be ready to go we will get started when Mr. 
Hall arrives. 

I have been informed that Mr. Hall is on his way, and we are 
going to assume that it is his pleasure that we do not hold you up 
any more than necessary, so we will go ahead and proceed. 

We don’t have control over when votes occur. Sorry to hold you 
up. This is an important hearing, and we do want to proceed. 

So at this time I will introduce our second panel of witnesses. Dr. 
Robert S. Dodd is the Safety Consultant and President of Dodd and 
Associates, LLC. Next, Dr. Jon Krosnick is the Frederic O. Glover 
Professor in Humanities and Social Science at Stanford University, 
and our last witness on this second panel is Dr. or rather, excuse 
me. Captain Terry McVenes, who is the Executive Air Safety 
Chairman of the Air Line Pilots Association. 

Welcome to all of you. As you know, we hope that you will sub- 
mit your full testimony and summarize it in five minutes if that 
is possible. If not, we do not want to interfere with a good hearing 
today. 

And so. Dr. Dodd, the microphone is yours. 

Panel 2: 

STATEMENT OF DR. ROBERT S. DODD, SAFETY CONSULTANT 
AND PRESIDENT, DODD & ASSOCIATES, LLC 

Dr. Dodd. Thank you. Good afternoon, Mr. Chairman and Mem- 
bers of the Committee. My name is Dr. Robert Dodd, and I appre- 
ciate the opportunity to address the Committee on the NAOMS 
Project. 

For seven years I served as the principal investigator for 
NAOMS. I consider myself extremely fortunate to have been in- 
volved with NAOMS. This was a unique project based on thorough 
preparation and outstanding science. 

NASA managers provided the research team with the support 
and leadership needed to design and conduct an exceptional 
project. The research team itself was composed of an extremely 
well-qualified and knowledgeable group of scientists whose commit- 
ment to the project was unparalleled. 

Finally and most importantly, I must acknowledge the commit- 
ment and effort of the hundreds of professional and general avia- 
tion pilots who helped design the survey and the 29,000 pilots who 
donated over 14,000 hours of their time to tell us about their safety 
experiences in an effort to improve the safety of the Nation’s air 
transportation system. 
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When I learned that NASA had decided the data collected by 
NAOMS would not be released to the public, I was disappointed 
and perplexed. I have seen many reasons cited for why NASA de- 
cided these data should not be released. The press reported that 
NASA was concerned that the data might frighten airline pas- 
sengers, and this would have a negative affect on the well being 
of the airlines. 

Other aviation organizations claim that the NAOMS data were 
soft data and voluntarily submitted. The implication was that the 
NAOMS data were somehow of limited or no value because they 
originated with pilots who were voluntarily responding to a survey. 

Finally, there are press reports that stated NAOMS data were 
not needed because current FAA oversight systems provided an 
adequate picture of the safety performance of the aviation system. 
I don’t agree with these perspectives. 

I believe the American public understands and accepts that trav- 
el by commercial airlines in the United States is the safest mode 
of travel in the world. Major air carrier crashes are thankfully rare 
events. I don’t believe based on my experience that the NAOMS 
data contained any information that would increase the passengers’ 
fear of flying. 

NAOMS data, which were collected to help insure that the U. S. 
airline safety remains best in the world, should be released so it 
can be used for its intended purpose. 

I would like to encourage the Committee to consider why a pro- 
gram like NAOMS is currently not operating. In most other aspects 
of public health and safety, U.S. Government and industry organi- 
zations routinely use surveys to identify and understand risks. 
Many of these programs have been in existence for years and are 
essential to effective oversight and evaluations of the Nation’s safe- 
ty and health programs. 

A program like NAOMS can help identify risks by obtaining in- 
formation from those who should know, the people operating the 
system. It can also help evaluate the safety impact of new tech- 
nologies as they are introduced. This is an important consideration 
in light of all the changes occurring in the aviation system on a 
daily basis and especially when we consider the new technolo^es 
such as the air traffic control overhaul, which is going to be coming 
shortly. 

Finally, an NAOMS-like program can provide quick insight into 
how well safety enhancements and improvements are working, a 
capability difficult to duplicate with today’s aviation safety over- 
sight systems. 

In closing, I believe that NAOMS should be restarted and oper- 
ated by an independent and unbiased organization. Such a pro- 
gram should receive funding directly from Congress to insure its 
budget remains adequate to fulfill its mission. 

I appreciate the opportunity to comment on this important pro- 
gram. 

[The prepared statement of Dr. Dodd follows:] 

Prepared Statement of Robert S. Dodd 

Good afternoon Mr. Chairman, Members of the Committee. My name is Dr. Rob- 
ert Dodd and I appreciate the opportunity to address the Committee on the National 
Aeronautics Operations Monitoring System, also known as NAOMS. 
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Between February 1998 and March 2005, a period of seven years, I served as the 
principal investigator for the NAOMS project. I participated in all aspects of the 
survey including its design, application, data analysis and project management, 
often in collaboration with Mr. Loren Rosenthal, the Battelle Project Manager for 
NAOMS. Battelle was the prime contractor for the project. 

I consider myself extremely fortunate to have been involved with NAOMS. This 
was a unique project based on thorough preparation and outstanding science. NASA 
managers provided the research team with the support and leadership needed to de- 
sign and conduct an absolutely outstanding project. The research team itself was 
composed of an extremely well qualified and knowledgeable group of scientists 
whose commitment to the project was unparalleled. Finally and most importantly, 
I must acknowledge the commitment and effort of the hundreds of professional and 
general aviation pilots who helped us design the survey and the 24,000 pilots who 
donated over 12,000 hours of their time to tell us about their safety experiences in 
an effort to improve the safety of the Nation’s air transportation system. 

I was disappointed and perplexed when I learned that NASA decided the data col- 
lected by the NAOMS survey would not be released to the public. While I know that 
the most notable denial was that issued to the Associated Press, the Johns Hopkins 
University Center for Injury Research and Policy, a reputable safety research orga- 
nization in addition to be a leading scholarly institution, was also denied. 

Many different reasons were cited for NASA’s refusal to release these data to the 
public. The press reported that NASA was concerned that the data might “frighten 
airline passengers” and this would have “a negative effect on the well being of the 
airlines.” Press reports also indicted that other aviation organizations claimed that 
the NAOMS data were “soft data” and voluntarily submitted. The implication was 
that the NAOMS data were somehow of limited, or no value, because they origi- 
nated with pilots voluntarily responding to a survey. Finally, there were press re- 
ports that stated NAOMS data were not needed because current FAA oversight sys- 
tems provided an adequate picture of the safety performance of the National Air- 
space System. 

I find these arguments without merit. 

I believe the American public understands and accepts that travel by commercial 
airlines in the United States is the safest mode of travel in the world. Major air 
carrier crashes are thankfully rare events. When a major crash occurs, it receives 
exceptional press coverage throughout the world, usually with images of destruction 
and chaos. Yet passengers continue to fly. I don’t believe that the NAOMS data con- 
tained any information that could compare with the image of a crashed air carrier 
airplane or would increase passengers’ fear of flying. 

I also don’t believe the argument that NAOMS data are somehow limited or of 
no value because they are derived from a survey has merit. All data used for anal- 
ysis, no matter its origin, have limitations and errors. Based on my experience, most 
if not all the databases used by the FAA for safety oversight and analysis contain 
errors and have limitations. This is why knowledgeable scientists and experts are 
involved in turning these data into useful information for decision makers. NAOMS 
data are no different in this regard. The NAOMS team made an extraordinary effort 
to clean and validate the data collected through the survey. The resulting data is 
of good quality and ready for meaningful analysis. Why would anyone decide that 
additional information, especially when it deals with the safety of the traveling pub- 
lic, should be hidden? 

Finally, the belief that the NAOMS data are not needed because current safety 
oversight systems are adequate is untrue. Not all airlines have Flight Operational 
Quality Assessment (FOQA) programs or participate in the Aviation Safety Action 
Program (ASAP), a pilot based voluntary reporting system. Further, current safety 
oversight systems do not do a good job of measuring safety errors in the general 
aviation fleet, among small commercial operators, or among maintenance techni- 
cians, all of which have a direct influence on airline safety. A program like NAOMS 
can provide a unique oversight capability for all of the aviation system. 

In closing I would like to encourage the Committee to consider why a program 
like NAOMS is not currently operating. In most other aspects of public health and 
safety, U.S. Government and industry organizations routinely use surveys to iden- 
tify and understand risks to public safety and health. Many of these programs have 
been in existence for years and are central to the evaluation and oversight of the 
Nation’s health and safety. 

A program like NAOMS can: 

1. Help identify risks before they result in losses by obtaining information from 
those who are in the best position to know, the people operating the system. 
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2. Help evaluate the impact of new technology, an important consideration in 
light of all the changes occurring in the National Airspace System including 
the overhaul of the air traffic control system. 

3. Provide quick insight into how well safety enhancements and improvements 
are working, a capability difficult to duplicate with today’s oversight systems. 

I believe NAOMS should be reinstituted and operated by an independent and un- 
biased organization. Such a program should receive funding directly from Congress 
to ensure its budget remains adequate to fulfill its mission. 

Thank you for the opportunity to comment on this important issue. 

Biography for Robert S. Dodd 


WORK EXPERIENCE 

Johns Hopkins University School of Public Health, Baltimore, MD; 1/2004- 
Present 

Adjunct Faculty 

I teach a course at the Johns Hopkins University Bloomberg School of Public 
Health titled Transportation Research, Public Policy and Politics. This is a graduate 
level course. This course is intended to provide an overview of the significant role 
of national politics on transportation safety policy in the United States. Using case 
studies of notable safety enhancement efforts in aviation, highway, rail and mari- 
time transportation, the students are introduced to the significant roles and inter- 
actions of lobbyists, industry associations, politicians, and federal agencies in trans- 
portation safety research and subsequent safety improvement rule-making. Through 
lectures, readings and a field trip, students learn that transportation safety and in- 
jury prevention improvements often require significant efforts to successfully navi- 
gate the path from research findings to interventions that improve the traveling 
public’s safety and health. 

Dodd & Associates, LLC, Gambrills, MD; 6/1998-Present 
Owner 

Dodd & Associates, LLC is a consulting company that specializes in transpor- 
tation safety research and analysis. As owner, 1 serve as the senior research sci- 
entist and manager. Our business focus includes transportation safety research, 
data analysis, research design, survey research, transportation injury control assess- 
ments, safety program design, safety training, safety audits and analysis, and 
OSHA compliance assessments. 

I serve as a research scientist on research projects for the Federal Government 
and private clients. In many of the projects, I have served as the principle investi- 
gator. Consequently, I am usually responsible for developing project proposals and 
the research protocol, project work plans and time lines, managing project partici- 
pants, writing the final reports and presenting the findings to the client and other 
organizations as required. I am knowledgeable about government contracting and 
grant procedures as a result my extensive experience in managing such programs 
both as a contract and grant recipient. 

A sample of projects include: 

Principal Investigator, National Aviation Operations Monitoring Service 
(NAOMS): Multi-year, multi-million dollar survey study that collected information 
on safety incidents from over 22,000 air line pilots and 4,000 small airplane pilots. 
Study was conducted for NASA. I oversaw experimental development, testing and 
application of the project research plan and survey. The surveys were conducted via 
telephone and achieved an 80 percent response rate. The project is now complete 
and papers are being written for peer review journals. 

Principal Investigator, Wide Area Augmentation System (WAAS): This research 
project was designed to quantify in dollars saved by the potential reduction in crash- 
es associated with the planned introduction of the wide area augmentation system 
(WAAS) navigation system. The WAAS is a satellite-based navigation system devel- 
oped by the Federal Aviation Administration (FAA) to provide precision approach 
capability to the majority of airports in the continental United States. The project 
was conducted for the FAA and resulted in a report for FAA use. 

Co-Principle Investigator, Evaluation of the Use of Common Denominators for 
Cross Modal Transportation Safety Evaluation: I served as a co-principal investi- 
gator with Professor Susan Baker on a Johns Hopkins University research project 
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to evaluate the feasibility of using common exposure measures for cross-modal eval- 
uations in transportation safety evaluations. This study was sponsored by the Bu- 
reau of Transportation Statistics which is part of the Department of Transportation. 

Audit Team Leader, Patient Transport System Operational Safety Audits: I lead 
a team of experts who evaluate the safety of patient transport operations (both 
ground and air) for medical transport services. We have completed over 65 audits 
to date. Focus of audits included patient safety, occupational safety and transport 
operations. 

Records Management Systems, Incorporated, Fairfax, VA; 3/1996-6/1998 
Senior Research Scientist 

I served as a senior research scientist for RMA, a government contractor sup- 
porting the Federal Aviation Administration’s (FAA) Office of Aviation Safety. I con- 
ducted safety research, assisted in the design of database and safety analysis sys- 
tems for the FAA’s National Aviation Safety Data Analysis System (NASDAC) and 
helped develop safety programs. I participated in strategic planning, helped design 
research protocols and project management plans, and participated in industry 
meetings for the FAA. 

A key component of NASDAC’s mission at that time was the evaluation and inte- 
gration of aviation data safety systems into a common access point for analysis. 
These data systems were owned and operated by the FAA, the National Transpor- 
tation Safety Board (NTSB), the National Aeronautics and Space Administration 
(NASA), the British Civil Aviation Authority (CAA), and private data sources such 
as AirClaims. As the primary analyst supporting the NASDAC’s mission, I became 
very familiar with these data sources. My familiarity originated with using these 
data for analytical projects and evaluating the databases for accuracy, structure, rel- 
evancy to current safety issues and much more. Through this experience, I became 
expert in the strengths and limitations of these data sets. 

Battelle Memorial Institute, Columbus, OH; 5/1990-3/1996 
Principal Research Scientist 

I supported Battelle’s transportation group conducting research and participating 
as a Battelle representative in meetings and conferences held in Washington D.C. 
I also supported the FAA’s Aviation Safety Reporting Program (ASRS), a voluntary 
aviation incident reporting system, by conducting analysis of the data contained in 
the ASRS database. I conducted analysis, generated reports, and presented findings 
of interest to both government and industry organizations. 

Johns Hopkins University School of Public Health, Baltimore, MD; 8/1988- 
5/1990 

Research Assistant 

I was a teaching and research assistant while a full-time doctoral student. As 
such, I assisted professors in the research activities conducing database design, de- 
velopment and research. I also assisted in teaching courses. 

National Transportation Safety Board, Washington, DC — 7/1986-8/1988 
Transportation Safety Specialist 

I was a transportation safety specialist and worked in the safety studies division. 
I was responsible for conducting targeted research investigations of specific trans- 
portation safety issues, writing summary reports and generating corrective rec- 
ommendations. I assisted in crash investigations and statistical evaluations. I also 
participated in industry meetings, wrote speeches for individual Board members and 
made public presentations. I left this position to return to school for my doctorate. 

Air Line Pilots Association, Herndon, VA; 6/1980-7/1986 
Staff Safety Engineer 

As a staff member of the Engineering and Air Safety Department, I supported 
pilot safety committees and worked on safety issues involving crash survival, airport 
design and airport safety. Part of my duties involved responding to FAA Notices of 
Proposed Rule-making (NPRM) for safety regulation rule changes. I also worked 
closely with the FAA and NTSB on a broad variety of air carrier safety issues. I 
also managed safety committees for the Association, participated in industry work- 
ing groups sponsored by the Society of Automotive Engineers, National Fire Protec- 
tion Association, American Association of Airport Executives and similar organiza- 
tions. 
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Freeway Airport Inc., Mitchellville, MD; 12/1978-6/1980 
Flight Instructor 

As a charter pilot and flight instructor I was responsible for conducting air taxi 
flights for customers and training primary, advanced and instrument pilots. 

EDUCATION 

Johns Hopkins University School of Public Health, Baltimore, MD; Doctorate, 5/ 
1992; Major: Public Health; Minor: Behavioral Science 

Relevant Course Work, Licensures and Certifications: 

This course of study was research-oriented and predominantly quantitative and 
lead to a Doctorate of Science (Sc.D). It included study of statistics, epidemiology, 
experimental design, survey design and application, database design, transportation 
safety and research methodology. The main focus was transportation injury preven- 
tion and occupational safety, with secondary study in the behavioral sciences. This 
focus included injury coding and outcome measurement, and observational study de- 
sign. My thesis evaluated occupant crash survival and was titled “Factors Related 
to Occupant Crash Survival in Emergency Medical Service Helicopters.” 

University of Southern California, Los Angeles, CA; Master’s Degree, 12/1981; 
Major: Safety 

Relevant Course Work, Licensures and Certifications: 

This degree program used an interdisciplinary systems approach to the theory 
and application of modern transportation safety practice. The curriculum included 
study in management, technology application, human factors, accident investigation, 
risk management, system safety, environment and communications. Focus areas for 
my specific course of study included: structural safety and failure analysis, accident 
investigation, human factors, system safety engineering, statistical analysis, and ex- 
perimental design in safety research. 

University of Maryland, College Park, MD; Bachelor’s Degree, 12/1978; 128 Semes- 
ter Hours; Major: General Studies 

Relevant Course Work, Licensures and Certifications: 

This course of study led to an independent studies degree with the main focus on 
the life sciences, including courses in micro biology, zoology, physiology, chemistry, 
and anatomy. 

AFFILIATIONS 

Association of Air Medical Services, Member, Board of Directors 
American Society of Safety Engineers, Professional Member 
American Public Health Association, Professional Member 

PROFESSIONAL PUBLICATIONS 

Scott A, Krosnick J, Dodd R, et ah, Comparing Telephone Interviews with Self- Ad- 
ministered Mailed Questionnaires: Results from a Field Experiment Assessing 
Reporting Accuracy. Public Opinion Quarterly, submitted. 

Baker S, Grabowski J, Dodd R, et ah, EMS Helicopter Crashes: What Influences 
Fatal Outcome? Annals of Emergency Medicine, April 2006 (Vol. 47, Issue 4, 
Pages 351-356). 

Enders J, Dodd R, Fickeisen F, Continuing Airworthiness Risk Evaluation, Flight 
Safety Digest, Flight Safety Foundation, Sept-Oct 1999, Arlington, VA. 

Enders J, Dodd R, et al., A Study of Airport Safety With Respect to Available Ap- 
proach and Landing Aids, Flight Safety Digest, Flight Safety Foundation, Nov. 
1995. 

Baker SP, Lamb M, Dodd R, Crashes of Instructional Flights, Analysis of Cases and 
Remedial Approaches, FAA Grant Report #93-G— 045, Johns Hopkins Center for 
Injury Research and Policy, Baltimore, MD, Oct. 1994. 

Dodd R, The Cost-Effectiveness of Air Medical Helicopter Crash Survival Enhance- 
ments, Air Medical Journal, 13:7, July 1994. 

Baker SP, Lamb MW, Li G, Dodd R, Human Factors in Crashes of Commuter Air- 
planes, Aviation Space and Environmental Medieine, 193, May; 64(5):417. 
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Dodd R, Occupant Survival In Emergency Medical Service Helieopter Crashes, 
Transportation Research Record of the National Research Council, 1992. 

Dodd R, ASRS: An Under used Resource, The Journal of Air Medical Transport, Vol. 
10, No. 10, Oct. 1991. 

Eldredge D, Dodd R, Mangold S, Categorization and Classification of Flight Man- 
agement System Incidents Report^ to The Aviation Safety Reporting System, 
Battelle Memorial Institute, Columbus, OH, Contract No. DRTS-57-89- 
D00086, June 1991. 

Dodd R, Reporting Accident Rates per 100,000 Patient Transports Responsible Tech- 
nique, letter to the editor. The Journal of Air Medieal Transport, Vol. 10, No. 
2, Feb. 

ADDITIONAL INFORMATION 

• Adjunct Faculty, Johns Hopkins University Bloomberg School of Public Health, 
Center for Injury Research and Evaluation 

• John W. Hill Safety Scholarship, University of Georgia 

• William Haddon Fellowship in Injury Control, Insurance Institute for Highway 
Safety 

• Graduate Research Award Program, Public-Sector Aviation Issues, Transportation 
Research Board, National Academy of Sciences 

• Outstanding Performance Award, National Transportation Safety Board 

• At-Large Member, Board of Directors, Association of Air Medical Services 

• Chair of the Safety Committee, Association of Air Medical Services 

• Airline transport rated multi-engine pilot (ATP-ME) 

Chairman Gordon. Thank you very much, Dr. Dodd. 

Our next witness, please proceed. 

STATEMENT OF DR. JON A. KROSNICK, FREDERIC O. GLOVER 
PROFESSOR IN HUMANITIES AND SOCIAL SCIENCES, STAN- 
FORD UNIVERSITY 

Dr. Krosnick. Thank you. Mr. Chairman, thank you for the op- 
portunity to testify today. I am a Professor at Stanford University 
with expertise in psychology and political science, and I have de- 
voted most of my career to the study and use of survey method- 
ology. I have conducted more than 100 surveys and have conducted 
research to identify best practices in the design of surveys. I have 
written more than 350 research papers and received 65 grants and 
contracts to support my research, mostly from the Federal Govern- 
ment. 

I have written a textbook in this area, and as an expert on sur- 
vey methods, I have advised many federal agencies on how to con- 
duct their surveys, including the GAO, the IRS, the CIA, the NIH, 
NOAA, EPA, the Census Bureau, the Bureau of Labor Statistics, 
CDC, and others. 

I am here to thank and congratulate NASA and to offer my 
praise to them for a job well done to the highest standards of excel- 
lence so far in their work on NAOMS. There are many data collec- 
tion systems in place to track air safety problems, and NAOMS is 
a terrific addition to this array. 

In my opinion NAOMS has been a great success, and NASA de- 
serves to be very proud of this success and deserves the thanks of 
this Congress and of all Americans. 

As you know NAOMS was designed to measure the frequency of 
the precursors of aviation accidents through statistically-reliable 
scientific surveys of pilots. You might imagine that information on 
these events can be collected reliably by machines, by black boxes 
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on aircraft, by computers in the air traffic control system, and by 
video cameras watching airport operations. 

But imagine the gigantic volume of information that would be 
collected by such systems in just one day and imagine trying to 
wade through that mountain of information to try to identify safety 
compromising events. And that mountain would not even include 
the many experiences and events that occur during interactions be- 
tween people without a machine record. 

This is why NAOMS was conceived as it was; to use the eyes and 
ears of the people actually operating the aviation system to track 
what they experience and convey the resulting information to pol- 
icy-makers. For decades the Federal Government has sponsored 
many longstanding and recurring survey projects to collect infor- 
mation used to promote public welfare. The unemployment rate is 
measured through surveys, the inflation rate is measured through 
surveys, and federal agencies regularly conduct surveys to measure 
much, much more. 

Surveys are a mainstay at the Federal Government and have 
been shown to provide valuable scientific measurements of the ex- 
periences of our nation’s huge population quickly, accurately, and 
inexpensively as compared to other ways to learn the same infor- 
mation. 

Loren Rosenthal’s vision of NAOMS is shown on this slide, which 
was presented by NASA in many public meetings. The NAOMS 
Project was to involve the design and implementation of surveys 
not only of pilots but also of air traffic controllers, flight attend- 
ants, and mechanics every week of every year to measure how 
many of various specific accident precursors they had witnessed 
while working during the past 60 days. 

As you can see from this diagram in the upper right, this was 
to be a permanent monitoring system. I was privileged to be asked 
to serve as a consultant to the team of superb professionals who 
have carried out the work done on NAOMS to date. As I watched 
the team do its work over a period of years, I saw a great deal 
about how it was done. 

I look forward to answering your questions, but in the remaining 
opening moments I have I would like to set the record straight on 
five important misunderstandings that have found their way into 
the public discussion of NAOMS during the past week. 

First, some people have claimed that the NAOMS methodology 
was not peer reviewed. This is incorrect. The survey methods used 
in NAOMS have been peer reviewed and widely accepted in the 
field for more than 40 years. And the NAOMS Team used peer re- 
viewed and well-established evaluation techniques to select the 
best standard methods for use in the NAOMS surveys. 

Furthermore, survey research experts at the White House Office 
of Management and Budget must review every federal survey 
project to assure that the methods to be used are optimal, and they 
reviewed and approved the NAOMS methodology. 

And prior to that approval process the NAOMS Team had held 
dozens of meetings, workshops, and consultations around the coun- 
try with aviation experts, interested parties, and social scientists to 
describe the project’s methodology and get reviews, comments, and 
suggestions. 
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Second, some people have said that NAOMS was not shut down 
prematurely. This is incorrect. The slide up on the screen shows 
you that initial NAOMS funding was intended to pay for surveys 
to he done not only of pilots but of air traffic controllers, flight at- 
tendants, and mechanics. But the funding for NAOMS was ended 
before that work was initiated. 

Third, some people have said that the NAOMS Project was de- 
signed simply to test the feasibility of a method, not to implement 
that method in a long-term survey monitoring system. This is in- 
correct. We determined that the method was viable and effective 
after a field trial involving 635 pilots. You don’t do 24,000 inter- 
views of pilots to test the feasibility of a method. You do that many 
interviews after you know the method is feasible and ready for 
prime time. 

Fourth, some people have said that if the NAOMS data were re- 
leased to the public, individual pilots or airlines would be identifi- 
able. This is incorrect. The overwhelming majority of NAOMS data 
cannot be linked to any pilot or airline because the system was set 
up to assure that from the start. The very small number of in- 
stances in which a pilot mentioned a specific airline or event date 
spontaneously can easily be removed from the public data set and 
made available to analysts only through Census Data Centers, 
which the Federal Government created exactly for the purpose of 
allowing researchers to use highly-confidential government data for 
research purposes while protecting anonymity. 

Lastly, some people have said NAOMS data cannot be used to 
compute the rates at which events happened because multiple re- 
spondents might have reported the same event, leading to overesti- 
mates. This is incorrect. NAOMS was designed intentionally to col- 
lect multiple reports of the same event, and NAOMS was also de- 
signed to implement a statistical procedure to recognize this mul- 
tiple reporting when translating the results of the surveys into 
computation of event counts. 

My best guess of why you heard earlier that events are — event 
rates are too high in the survey is because that correction is not 
being implemented properly. 

Thus, these five criticisms of NAOMS are unfounded, and for 
these many reasons I believe that NASA deserves terrific praise for 
initiating NAOMS and for carrying out the work done so far so 
well. The method offers a new way to complement existing streams 
of data on aviation safety and it is relatively cheap and quick com- 
pared to the other methods being implemented. 

So in closing I want to thank NASA for the decision to make ex- 
isting NAOMS data available to the public, along with complete 
documentation on exactly how the data were collected, but most 
importantly I want to urge NASA and this committee to restart 
NAOMS data collection where they left off. There is much left on 
the diagram on the screen to be done, and if NASA gets to work 
doing it, there will almost certainly be terrific benefits for this na- 
tion. And this committee can take some credit for those benefits if 
it comes about. 

NASA did a great job with NAOMS already, and they have a 
unique position of trust, objectivity, and scientific expertise in the 
aviation world that will allow them to carry out this work with effi- 
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ciency and credibility. I hope they will chose to continue this impor- 
tant work in the future. 

Thank you very much. 

[The prepared statement of Dr. Krosnick follows:] 

Prepared Statement of Jon A. Krosnick 

Thank you very much for the invitation to submit this statement and to testify 
before the Committee as it explores the history of NASA’s National Aviation Oper- 
ations Monitoring Service (NAOMS). 

Currently at Stanford, I am the Frederic O. Glover Professor of Humanities and 
Social Sciences, Professor of Communication, Professor of Political Science, Professor 
of Psychology (by courtesy), and Associate Director of the Institute for Research in 
the Social Sciences. 

As a member of the team that developed NAOMS, my role was as an expert on 
survey research methodology and questionnaire design. 

My Qualifications and Experience 

While I have been a Professor at the Ohio State University and now at Stanford 
University, a great deal of my research has involved the collection and analysis of 
survey data, and many of my publications have been designed to identify best prac- 
tices in survey methodology. 

As my curriculum vitae outlines (see Appendix A of this statement), I have pub- 
lished five books and am currently completing the fifth, The Handbook of Question- 
naire Design (Oxford University Press). I have published 107 journal articles and 
book chapters in peer-reviewed publications. I have presented 252 papers reporting 
my research findings at research conferences around the world, where presentations 
were selected through a peer review process. I have received 65 grants and contracts 
supporting my research and am currently overseeing active grants and contracts to- 
taling more tban $10 million. 

I have served as a consultant to the following federal agencies on survey research 
issues: The Government Accountability Office (GAO), tbe Internal Revenue Service 
(IRS), tbe Central Intelligence Agency (CIA), tbe National Institutes of Health 
(NIH), tbe National Oceanic and Atmospheric Administration (NOAA), the Environ- 
mental Protection Agency (EPA), the Bureau of the Census, the Bureau of Labor 
Statistics (BLS), the Centers for Disease Control and Prevention (CDC), and the Na- 
tional Cancer Institute (NCI). I have advised these agencies on how to implement 
best practices in the survey research they conduct. 

I currently serve as co-principal investigator of the American National Election 
Study (ANES), the academic world’s leading survey study of voting and elections, 
which is supported by a $7.6 million grant from the National Science Foundation. 
This project began in 1948 with a national survey of a representative sample of 
American voters, and the same sort of survey has been conducted every two years 
since then. The data from the ANES are made public at no charge to all interested 
investigators around the world. As co-principal investigator, my responsibilities in- 
clude all decisions about methodology for the collection of the survey data and all 
decisions regarding the design of the questionnaires used. 

I also serve on the Board of Overseers of the General Social Survey, which is the 
Nation’s preeminent survey study of trends in Americans’ social and political atti- 
tudes and behavioral experiences. Since the early 1970s, this study has involved an- 
nual or biannual surveys of representative national samples of American adults 
interviewed in their homes for hours and documenting a wide range of their opin- 
ions and experiences. Like the ANES, the CSS has been funded by the National 
Science Foundation, and the study’s data are made available for free to all inter- 
ested researchers around the world and 

The NAOMS Vision 

The instigation of NAOMS was a commitment made in the 1990s by the Federal 
Government to reduce the risk of commercial airplane crashes by a specific targeted 
amount within ten years. Once that target was set, federal agencies looked for ways 
to assess whether that goal would be achieved and realized they had none. Simply 
tracking plane crashes would not be sufficient, because they happen extremely rare- 
ly and therefore do not indicate the amount of underlying risk posed by the many 
small events that, when cumulated, can increase the risk of an accident. Con- 
sequently, some alternative monitoring system was needed. 

The Federal Aviation Administration, other agencies, and private sector organiza- 
tions (e.g., commercial airlines) have been collecting some information on the fre- 
quency with which some risk-elevating events have been occurring. But the array 
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of event types being tracked was more limited than is needed for thoroughly track- 
ing the functioning of the entire air travel system. Some anecdotal information has 
also been collected, but this information could not he used to calculate statistically 
reliable risk levels. Therefore, a new system for collecting information on the fre- 
quency of precursors to accidents was needed. 

NAOMS was designed to serve this purpose and to collect the needed information 
via high quality scientific and reliable surveys of people around the world who were 
watching the operation of the aviation system first-hand and who knew what was 
happening in the field. Indeed this use of the survey method was in keeping with 
many other long-term federally funded survey projects that provide valuable infor- 
mation to monitor public risk, identify sources of risk that could be minimized, iden- 
tify upward or downward trends in specific risk areas, to call attention to successes, 
identify areas needing improvement, and thereby save lives while promoting com- 
merce in the Nation. 

As originally conceived by Battelle Project Manager Loren Rosenthal, NAOMS 
was to be a multifaceted survey project building on the Aviation Safety Reporting 
System (ASRS). For many years, ASRS has been a successful system for collecting 
anecdotal information from pilots about some of the risk-elevating events they wit- 
nessed. Each time an event occurs, a pilot can choose to fill out a form describing 
it briefly and mail the form to NASA’s ASRS office in Mountain View, California. 
An aviation expert then telephones the reporter to conduct a telephone interview 
to gather detailed information about the event. A subset of this information is then 
entered anonymously into a database that NASA maintains. And when important 
insights about risks have been obtained through this system, NASA has sent out 
reports to the aviation community. 

ASRS has successfully collected information that has had observable positive ef- 
fects enhancing public safety. Pilots have come to trust it and NASA generally (be- 
cause nothing undesirable has occurred to a pilot as the result of filing an ASRS 
report), and ASRS has had the flexibility to collect data on whatever events pilots 
deem worth reporting. 

But this flexibility also constitutes a significant limitation of ASRS as well. Be- 
cause pilots voluntarily choose to file reports on events, their choices about when 
to report and what to report are uncontrolled. Consequently, many safety-related 
events go unreported to ASRS. And as a result, it is impossible to use ASRS to track 
trends in event rates over time. Therefore, NAOMS was envisioned to complement 
ASRS by producing accurate measurements of rates and trends in rates of a wide 
array of types of events. 

Every week of every year, NAOMS was planned to collect information from a rep- 
resentative sample of pilots flying commercial aircraft. The pilots would be asked 
to report the number of each of a series of different specific events that they had 
witnessed during a specific recent time period (e.g., the last 60 days). These counts 
could then be used to calculate the rates at which the events had occurred during 
that period throughout the entire air travel system. 

NAOMS had the potential to succeed especially because ASRS had already been 
successful. The trust that the community of commercial pilots had developed in 
NASA through its running of ASRS meant that these pilots could most likely be 
counted on to participate in NAOMS surveys at a high rate without concern about 
retribution. That is, the pilots could be expected to provide accurate and honest re- 
ports of event frequencies, because they already knew that NASA (through ASRS) 
was capable of compiling and reporting such data in a trustworthy and safety-en- 
hancing way. 

But NAOMS was envisioned to go well beyond ASRS, by tapping the knowledge 
and experiences of other professionals participating in the air travel system and ob- 
serving risk-elevating events. Specifically, the original plan for NAOMS included 
collecting survey data every week of every year from general aviation pilots, heli- 
copter pilots, air traffic controllers, flight attendants, and mechanics, as shown in 
the following timeline that was presented by NASA at various public meetings de- 
scribing the project: 
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Thus, the plan was to design and implement a “permanent survey” data collection 
operation to generate ongoing data to track event rates into the future. 

NAOMS Resembled Many Other Federal Surveys 

This use of survey methodology in NAOMS was consistent with the conduct of 
surveys by many organizations in the public and private sectors to track rates of 
events over time and to inform decision-making and organizational practices. Survey 
methodology is a highly developed science that can utilize reports of people’s experi- 
ences to document events occurring around the Nation and around the world quickly 
and cheaply. In fact, each year, billions of dollars are spent conducting surveys 
around the world. The U.S. Federal Government is one of the largest producers of 
such data. For decades, survey data have been routinely collected and used by many 
federal agencies to track contemporary life in America in a wide array of domains 
and to provide valuable information for policy-making and policy implementation. 

A small subset of the survey research projects that have been funded by the U.S. 
government continuously, beginning in the years shown and sponsored by the agen- 
cies in parentheses, includes: 

• Survey of Income and Program Participation (Census Bureau) 1984- 

• Consumer Expenditure Surveys (Census Bureau) 1968- 

• Annual Housing Surveys (Census Bureau) 1973- 

• Survey of Consumer Attitudes and Behavior (National Science Foundation) 
1953- 

• Health and Nutrition Examination Surveys (National Center for Health Sta- 
tistics) 1959- 

• National Health Interview Surveys (National Center for Health Statistics) 
1970- 

American National Election Studies (National Science Foundation) 1948- 

• Panel Study of Income Dynamics (National Science Foundation) 1968- 

• General Social Survey (National Science Foundation) 1972- 

• National Longitudinal Survey (Bureau of Labor Statistics) 1964- 

• Behavioral Risk Factor Surveillance System (Centers for Disease Control and 
Prevention) 1984- 

• Monitoring the Future (National Institute of Drug Abuse) 1975- 
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• Continuing Survey of Food Intake by Individuals (Department of Agriculture) 
1985- 

• National Aviation Operations Monitoring System (National Aeronautics and 
Space Administration) 2002- 

• National Survey of Drinking and Driving (National Highway Traffic Safety 
Administration) 1991- 

• National Survey of Family Growth (National Center for Health Statistics) 
1973- 

• National Survey of Fishing, Hunting, and Wildlife-Associated Recreation 
(Census Bureau) 1991- 

• National Survey of Child and Adolescent Well-Being (Department of Health 
and Human Services) 1997- 

• Survey of Earned Doctorates (Science Resources Statistics Program, National 
Science Foundation) 1958- 

• National Survey on Drug Use and Health (Department of Health and Human 
Services) 1971- 

• Youth Risk Behavior Surveillance System (Department of Health and Human 
Services) 1990- 

• National Crime Victimization Survey (Bureau of Justice Statistics) 1973- 

• Schools and Staffing Survey (National Center for Educational Statistics) 
1987- 

• Educational Longitudinal Survey (National Center for Educational Statistics) 
2002- 

• Current Emplo 3 unent Statistics Survey (Bureau of Labor Statistics) 1939- 

Just a few of the many other major surveys sponsored by federal agencies over 
the years include: 

• National Survey of Distracted and Drowsy Driving (National Highway Traffic 
Safety Administration) 

• National Survey of Veterans (Department of Veterans Affairs) 

• National Survey of Children’s Health (Health Resources and Services Admin- 
istration’s Maternal and Child Health Bureau) 

• National Survey of Recent College Graduates (Science Resources Statistics 
Program, National Science Foundation) 

• National Survey of Speeding and Other Unsafe Driving Actions (National 
Highway Traffic Safety Administration, Department of Transportation) 

Survey data form the basis of many important government policy-making deci- 
sions. For example, economists in the Federal Reserve and other agencies pay close 
attention to the federal unemployment and inflation rates, both of which are cal- 
culated using data from national surveys. The many other federal agencies listed 
above collect survey data because those data are used in on-going decision-making. 

Decades of research have shown that the reliability and validity of optimally-col- 
lected survey data are generally quite high, and that respondents can be relied upon 
to provide quite accurate descriptions of their past experiences, behaviors, and opin- 
ions. Most visibly, surveys conducted just before U.S. presidential elections predict 
the actual election vote results very closely (see, e.g., Visser, P.S., Krosnick, J.A., 
Marquette, J., & Curtin, M., 1996; Mail surveys for election forecasting? An evalua- 
tion of the Columbus Dispatch poll. Public Opinion Quarterly, 60, 181-227, Visser, 
P.S., Krosnick, J.A., Marquette, J., & Curtin, M., 2000; Improving election fore- 
casting: Allocation of undecided respondents, identification of likely voters, and re- 
sponse order effects. In P. Lavrakas & M. Traugott (Eds.), Election polls, the news 
media, and democracy. New York, NY: Chatham House). Even when there is error 
in such survey measurements (and there is), the error is not huge in percentage 
point terms (bearing in mind that a small shift in percentages can change the win- 
ner of a close election). For example, since 1936, the percent of votes won by the 
winner has correlated with the Gallup Poll’s pre-election prediction of that percent- 
age .85, a nearly perfect association. ^ Likewise, since 1948, the American National 
Election Study surveys’ post-election measurements of the proportions of votes won 


1 Correlations can range from 1 (meaning a perfect match between the variables) to 0 (mean- 
ing a relation between the variables no better than chance) to -1 (meaning a perfect inverse rela- 
tion between the variables). 
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by the winning presidential candidate have correlated with official government vote 
counts .92, again nearly perfect. 

Equally striking are the results of the Monthly Survey of Consumer Attitudes and 
Behavior, conducted continuously hy the University of Michigan’s Survey Research 
Center since 1970. Each month, a representative national sample of American 
adults has heen asked what they expect to happen to the unemployment and infla- 
tion rates in the future (as well as many other topics), and their aggregated answers 
have predicted later changes in actual unemployment and inflation remarkably well 
(correlations of .80 and .90, respectively, between 1970 and 1995). This is testimony 
not only to the aggregated wisdom of the American public but also to the ability 
of scientific surveys to measure that wisdom accurately. 

A high level of accuracy can be achieved if optimal procedures are implemented 
to conduct a survey, and departures from such procedures can significantly com- 
promise the accuracy of a survey’s findings. Necessary features include drawing a 
representative sample of the population, taking extensive steps to collect data from 
as many sampled people as possible, optimizing the choice of survey mode to achieve 
accurate measurements, asking questions that are easily comprehensible and do not 
entail biased wording or format, weighting results to correct for unequal sampling 
probabilities, and much more. 

Survey Methods Development in NAOMS 

When I was brought onto the research team, I was told that the project was com- 
mitted not just to designing and conducting surveys, but to doing so with the best 
possible practices to assure the most accurate data possible. Thus, rather than sim- 
ply using intuition and budget limitations as guidelines for making methodological 
decisions, the project set out to design practices that would optimize data accuracy. 

To this end, we conducted a series of studies, including a large-scale field trial, 
to answer a series of questions with regard to the first survey we developed for air 
carrier pilots: 

• What risk-elevating events should we ask the pilots to count? 

• How shall we gather the information from pilots — written questionnaires, 
telephone interviews, or face-to-face interviews? 

• How far back in the past can we ask pilots to remember without reducing the 
accuracy of their recollections? 

• In what order should the events be asked about in the questionnaire? 

What events? The goal of the NAOMS survey was to collect information on as 
many different sorts of risk-elevating events as possible. To begin generating a com- 
prehensive list of such events, we conducted a series of focus group discussions with 
professionals who were active in the air traffic system, including air carrier pilots, 
general aviation pilots, helicopter pilots, and air traffic controllers. In each of these 
group discussions, we asked participants to generate as comprehensive a list of risk- 
inducing events as they could during a two-hour period. These exercises revealed 
a coherent and repeatedly-occurring list of events that seemed quite suitable for 
tracking by NAOMS surveys. 

In addition, we consulted with industry and government safety groups, including 
members of CAST, the FAA, and the analysts who conducted telephone interviews 
of pilots submitting reports to ASRS. We also reviewed the contents of aviation 
event databases, such as the ASRS, NAIMS, and BTS databases. In the end, we 
chose to track a set of events that was faithful to those pinpointed by these data- 
gathering exercises. 

What mode? At the time that NAOMS was launched, it was widely recognized in 
the survey research community that face-to-face interviewing was the optimal way 
to collect accurate and honest data from respondents. Although most surveys at that 
time were being conducted by telephone, the Federal Government’s most important 
and visible surveys continued to rely on face-to-face interviewing. When a com- 
petent, committed, and professional interviewer meets face-to-face with a respond- 
ent, the respondent develops a sense of trust in and rapport with the interviewer, 
inspiring the respondent to devote the cognitive effort needed to generate accurate 
responses and the confidence that his/her identity will be protected, so that honest 
reports can be provided without fear of retribution. 

We therefore decided to explore the viability of face-to-face interviewing of pilots 
for NAOMS. However, we recognized that such interviewing would be costly and 
logistically challenging, so we also explored the viability of two alternative modes: 
telephone interviewing and paper-and-pencil questionnaires. At the time we initi- 
ated NAOMS, the published survey methodology literature did not offer clear guid- 
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ance about the quality of data to be expected from these two latter modes. We there- 
fore designed a “field trial” to compare the three modes of data collection. 

At the start of the field trial, a sample of licensed pilots was selected to be inter- 
viewed face-to-face. But it quickly became clear that because of the ongoing mobility 
of the pilots, it would be practically impossible to coordinate schedules with them 
to allow interviewers to meet with them and conduct interviews at anything ap- 
proaching a reasonable cost. Therefore, face-to-face interviewing was abandoned. 
Consequently, the field trial focused on comparing telephone interviewing and paper 
questionnaires mailed to respondents using a method developed by Professor Don 
Dillman (a long-time consultant to the U.S. Census Bureau) to assure high response 
rates. 

Pilots were randomly assigned to be interviewed in one of these modes, and the 
survey research group at Battelle’s Center for Public Health Research and Evalua- 
tion conducted the data collection. The cost per interview was $60 for each mailed 
questionnaire completed, as compared to $75 for each telephone interview com- 
pleted. But according to all indicators of data quality, we got what we paid for: the 
telephone interviews yielded superior data. For example, the response rate for the 
mail questionnaires was 73 percent, and the response rate for the telephone inter- 
views was 81 percent. Whereas pilots never failed to answer a question during a 
telephone interview, respondents failed to answer 4.8 percent of the questions on 
the paper questionnaires. Respondents reported significantly more confidence in the 
accuracy of their answers during the telephone interviews than of their answers on 
the paper questionnaires. And a built in accuracy check showed that the telephone 
responses were 30 percent more accurate than the paper responses. We therefore 
chose to conduct the survey via telephone interviews. 

How far back in the past could pilots remember accurately^ Our goal was to collect 
information on as many events as possible without compromising the accuracy of 
recollections. The longer the time period that pilots were asked to describe, the more 
rare events could be detected, with no added cost. But if the recall period addressed 
in the questionnaire was short, then we would have had to increase the number of 
pilots interviewed considerably in order to detect rare events. A comprehensive re- 
view of the existing scholarly literature did not provide clear guidance on what the 
optimal recall period would be for NAOMS pilots, so we built into the field trial a 
manipulation designed to identify this optimal recall period. 

Specifically, we randomly assigned some pilots to report on the events they wit- 
nessed during the last week and others to report on the last two weeks, the last 
four weeks, the last two months, the last four months, or the last six months. We 
found that the most accurate reports were provided for the two-month recall period, 
so we selected that period for the final questionnaire. During the initial months of 
NAOMS main study data collection, respondents were randomly assigned to be 
asked about either the last 30 days, the last 60 days, or the last 90 days. But even- 
tually, all pilots were asked about the last 60 days. 

What order of questions? Once we had specified a list of events to be addressed, 
we had to specific the order in which to ask about these events. If the order is opti- 
mized, it can make respondents’ reporting process easier and their reports more ac- 
curate. And if order is not optimized, it can increase the difficulty of the task for 
the respondents, decrease their enjoyment of it, thereby decrease their motivation 
to provide accurate reports, and in the end, reduce the accuracy of the reports they 
do provide. 

Optimizing question order begins with the recognition that more complete and ac- 
curate recollection occurs when question order matches the way that information is 
organized in people’s long-term memories. That is, psychologists believe that clus- 
ters of related pieces of information are stored together in memory. Asking a person 
to go to a specific location in memory and retrieve all the needed information there 
before moving on to retrieving information from a different location is preferable to 
asking people to jump around from place to place in memory, question by question 
(e.g., Barsalou, 1988; DeNisi & Peters, 1996; Raaijmakers, & Shiffrin, 1981; 
Sudman, Bradburn, & Schwarz, 1996; Tulving, 1972). 

According to this logic, memories of similar safety-compromising events are likely 
to be stored together in clusters in pilots’ memories. So once a pilot begins retrieving 
memories from a particular cluster, it is easiest and most efficient to recall all other 
memories in that cluster, rather than jumping to another cluster. Therefore, our 
questionnaire grouped together questions asking about events that were stored near 
one another in pilots’ memories. 

Identifying each respondent’s memory organization scheme at the start of each 
interview is not practical. However, it was possible to assess the most common type 
or types of mental organizations used by pilots and tailor our questionnaire design 
to those types. We conducted a series of studies using a series of methods drawn 



58 


from cognitive psychology to identify pilots’ memory organizations, and the results 
of these studies clearly pointed to a memory organization that applied well across 
pilots and that we showed could be used to enhance the accuracy of recollections. 
In fact, our testing indicated that using the memory organization we identified to 
order questions enhanced recall accuracy by 25 percent or more over other orders 
we tested. 

Questionnaire pretesting. Once a survey questionnaire is designed, it is important 
to pretest it in various ways to assure that respondents understand the questions 
and can answer them. To test understandability and answerability, we conducted 
a series of tests. One test was built into the field trial, whereby we asked respond- 
ents to comment on and evaluate the understandability of the questions and to iden- 
tify any questions that were not sufficiently clear and understandable. We also con- 
ducted cognitive think-aloud pretest interviews using a technique pioneered by re- 
searchers at the National Center for Health Statistics. This involved having pilots 
listen to the questions, restate them in their own words, and think aloud while an- 
swering the questions. These pretests were used to identify instances in which ques- 
tion wording needed improvement. 

Field trial results. The field trial involved collecting data from about 600 pilots, 
and this allowed us to evaluate the performance of the methodology fully. The re- 
sults produced by the field trial documented that the methodology worked well. We 
achieved a very high response rate, and tests indicated high validity of the data. 
Thus, at the conclusion of the field trial, we had evidence sufficient to conclude that 
the method was well-designed and suitable for generating reliable data. 

Peer reviewing. Questions have been raised recently about whether the NAOMS 
methodology was subjected to a peer review process. In fact, peer review did occur. 
The research plan for NAOMS was presented at many public meetings and private 
meetings with stakeholder organizations and with experts involved in aviation and 
social science researchers. In all of these meetings, details of the rational for 
NAOMS and its methodology were described. The attendees asked questions, made 
comments, and offered suggestions. In addition, multiple meetings were held with 
large groups of NASA staff and FAA staff to provide details on the NAOMS plan 
and accomplishments and to acquire feedback. 

As far as I understand, NASA did not request or suggest to the NAOMS project 
team that any additional peer review occur. If such a request had been made, we 
would have been happy to implement additional review processes. However, that 
lack of such a request was not surprising to me or unusual in the context of federal 
survey design and data collection. I have been involved in many federal survey 
projects, and I have advised federal agencies on many others. The vast majority of 
these projects involved less peer review than NAOMS carried out. In fact, the only 
federally funded survey studies I know of that have routinely involved elaborately 
structure peer review processes are ones that were conducted by the government for 
use in litigation. These peer review processes rarely yielded significant changes in 
the survey process. I therefore do not believe that any additional peer review of the 
NAOMS methodology would have been significantly beneficial or caused any signifi- 
cant changes in procedure. 

An important reason for this is that in my role as a professor, I am responsible 
for keeping fully informed about the state of the survey methodology literature and 
new developments in survey techniques. By reading printed and online publications 
and attending conferences to hear presentations, I stay abreast of the field’s under- 
standing of best practices. Consequently, I was called upon regularly to evaluate our 
methodology vis-a-vis common practices in the field of survey research and the 
views of my professional peers on design issues. Thus, the views of my peers were 
regularly a focus during our planning process. 

Summary. The methods we used to develop the NAOMS questionnaire were state 
of the art. Indeed, the preliminary studies we conducted constitute valuable con- 
tributions to the scholarly literature on optimal survey design, producing findings 
pointing to best practices and identifying new methods for future tests intended to 
optimize survey designs. Thus, NASA can be very proud of what it accomplished 
during this phase of the project. 

My View of NAOMS 

It was a privilege and an honor for me to have been asked to serve as a method- 
ology expert on the NAOMS project. And it was a pleasure to work with the re- 
search team that carried out the project. Robert Dodd (now of the NTSB), Loren 
Rosenthal and Joan Cwi (of Battelle Memorial Institute), and Mary Conners and 
Linda Connell (of NASA) were consummate professionals who worked wonderfully 
together, even through times of tough decision-making. And the work done by the 
team was of superb quality. 
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Because NAOMS was so well conceived, I looked forward to continuation of the 
project and the development of a large publicly available database for the study of 
air travel safety. In our public meetings with interested parties, we presented the 
following slides to illustrate the widespread use of surveys by federal agencies and 
the common practices for running these surveys over long time periods and distrib- 
uting the data. 
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Thus, we intended to set up such a long-term data collection and dissemination 
system for NAOMS. 

When I heard that interviewing of air carrier pilots had been terminated and then 
that all funding for NAOMS had been stopped, I was surprised. As far as I knew, 
the project had been conducted according to best practices, and nothing that hap- 
pened during that period suggested anything to the contrary. 

In my view, NAOMS was intelligently conceived and excellently implemented. 
Thus, for as far as it went, NAOMS deserves a great deal of praise from NASA and 
from all Americans. Indeed, NASA and the Federal Government should be very 
proud of what it accomplished with NAOMS, because its success is just what all 
government agencies hope for when setting out to do good for this nation. 

My belief in the value of NAOMS for this country led me to write an op-ed essay 
published in the New York Times in 2006 just after I got the news of discontinued 
funding. I wrote that essay with the goal of calling attention to the great success 
of NAOMS and perhaps to lead to a reconsideration of its termination. 

At the very least, I hoped that a way could be devised to allow researchers to have 
access to the data that were collected via approximately 24,000 interviews with air 
carrier pilots over a period of years. 

These data can be useful in a number of ways. First, they can document the fre- 
quency with which various types of events were occurring. According to our inter- 
views with pilots early on in the project, they thought that NAOMS would be valu- 
able partly because it would call attention to surprisingly high frequencies of some 
low-risk events that could be easily reduced or eliminated. 

Second, the NAOMS data can be compared to data on the frequency of similar 
events collected by other data sources. For example, ASRS and the FAA collect data 
that can be used to compute event rates and compared directly to some of the events 
asked about in the NAOMS questionnaire. If the NAOMS questionnaires yield dif- 
ferent rates than these other reporting systems, that would highlight potential op- 
portunities to explore the sources of those discrepancies, which might yield improve- 
ments in measurement methods and a clearer understanding of what measurement 
procedures are most accurate. 

Third, the NAOMS data can be used to compute trends over time in event rates. 
This was of course the primary intended purpose of NAOMS when it was originally 
envisioned. Thus, NAOMS could be used to gauge whether changes in the air travel 
system during the years of data collection were successful in reducing risk. Because 
NAOMS data were collected both before and after September 11, 2001, it would be 
possible to see how the changes in practices that occurred at that time translated 
into changes in event frequencies. 

Fourth, the NAOMS questionnaires are designed in ways that allow analysts to 
assess some of the conditions under which particular types of events are most likely 
to occur. For example, it is possible to explore whether some types of events oc- 
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curred more on aircraft flown by pilots with less total career flying experience or 
by pilots with more than a certain amount of experience. It is possible to explore 
whether some types of events occurred more on some types of aircraft than on oth- 
ers. Such findings could be used to inspire further research to identify the reasons 
for the observed relations and then perhaps to change aviation practices to enhance 
safety. 

Fifth, the NAOMS data would allow researchers to conduct studies for optimizing 
survey methods generally. Not only is this possible by publishing reports of the field 
trial and preliminary studies done to prepare the NAOMS questionnaire and meth- 
odology, but the main study data can he used for this purpose in multiple ways. For 
example, it would be possible to compare the findings of data collected from pilots 
asked about events they witnessed during the last 30, 60, or 90 days to see how 
length of the recall period affected the accuracy of their recollections. This would 
be useful information to inform survey designers generally interested in optimizing 
recall questions. Also, it would be possible to explore how survey non-response is 
related to survey results, addressing a particularly hot topic in the survey method- 
ology literature at the moment. 

For all of these reasons, I believe that the existing NAOMS data should 
be made publicly available right away so that analysts can learn everything 
that can be learned from the data, to make the most of the $8.4 million that NASA 
spent on the project. I believe that the model for making these data public should 
be the ASRS. NASA has been very successful in setting up a system for fully pub- 
licly disseminating the terrifically valuable information provided by pilots through 
the ASRS reporting system, and a comparable dissemination system can be created 
for NAOMS data as well. 

Documenting the NAOMS Data in Detail 

In order to allow the dissemination of these data to yield the most positive bene- 
fits, it is essential that NASA provide extensive and detailed documentation of the 
procedures by which the study was designed and the procedures by which the main 
data were collected. This includes descriptions of sampling, of respondent recruiting, 
of locating potential respondents, of training interviewers, of releasing cases for 
interviewing at particular times, and more. The full array of electronic files docu- 
menting all phases of the data collection should be made public while protecting the 
identities of the individuals who were interviewed. 

In addition, NASA should help analysts use the data by providing written guide- 
lines on how to properly analyze the data in light of the study design. No one knows 
the design complexities better than the NAOMS research staff. So they should write 
documentation to help analysts understand the origins of and potential uses of the 
data set. 

Just one illustration of how complex analysis of these data is involves the issue 
of multiple reporting of the same event. One potential use of NAOMS data is to cal- 
culate the rates at which particular risk-increasing events happened during par- 
ticular time periods. NAOMS was designed to 3deld such estimates, but calculation 
of them must be done carefully. 

Consider, for example, bird strikes. An analyst might be tempted to simply count 
up the number of times that pilots who were interviewed during a particular time 
period (e.g., calendar year 2003) reported experiencing a bird strike. Then, the ana- 
lyst might be tempted to multiply this total by the ratio of the total number of li- 
censed pilots during that time period divided by the number of pilots who completed 
interviews in the survey to yield a projected total number of bird strikes that oc- 
curred to the entire population of pilots. 

However, multiple pilots witnessed each bird strike, and each bird strike could 
have been reported by each of those pilots. Specifically, a collision of a bird with 
an airplane would have been witnessed by two pilots on aircraft with two cockpit 
crew members and by three pilots on aircraft with three cockpit crew members. 
Thus, each bird strike had twice the probability of being reported by two-crew air- 
craft pilots and three times the probability of being reported by three-crew aircraft 
pilots. So in order to calculate the number of events accurately, the observed total 
number of events must be adjusted downward to account for this multiple reporting. 

NAOMS was designed knowing that this sort of calculation must be carried out. 
The questionnaire collected information necessary to implement corrections for this 
multiple reporting. Providing information to analysts about how to do this computa- 
tion would be a valuable public service. With substantial documentation accom- 
panying the data, analysts can be sure to calculate statistics properly by taking into 
account all such analytic considerations. 

In addition to providing this documentation immediately, I would strongly rec- 
ommend to NASA that they assemble and fund a “swat” team of suitable experts 



63 


to conduct all possible analyses with the NAOMS data and issue an initial report 
of their findings as quickly as possible. Subsequent reports can then be issued later 
as additional analyses are conducted. 

I assume that this “swat team’s” effort should build on the work that NASA has 
done already in constructing a final report on the data, which they planned to re- 
lease later this year. I have not seen a draft of that report and don’t know anything 
about its contents. But if it is not completely comprehensive in addressing all issues 
that the data can address and completely comprehensive in fully documenting all 
procedural details of how the data were collected, I would recommend that its scope 
be expanded accordingly, with proper government funding to permit it to be done 
as well as all of the rest of NAOMS to date. 

The Future of NAOMS 

One might imagine that the book has been closed on NAOMS and that clean-up 
activity is all that remains on this project. But I believe that to think of NAOMS 
in these terms would forego a wonderful opportunity for NASA and for this govern- 
ment and for this country. 

NAOMS data are not being generated by any other source. And from all indica- 
tions, the NAOMS data that were collected are reliable and valid. Furthermore, our 
team’s public meetings with stakeholders indicated considerable enthusiasm for the 
sorts of data that NAOMS was intended to provide. 

Therefore, I believe, the vision of a multi-faceted NAOMS data collection moni- 
toring service was and is terrifically positive for everyone who flies on planes, every- 
one who works in the commercial aviation system, everyone who manufactures air- 
planes, and everyone who monitors and helps to optimize aeronautics in American. 

Consequently, I recommend restarting NAOMS data collection where it left off 
and bring its potential fully into being. Doing so would be a great service of this 
government to this country. 

There has been some discussion recently of the notion that NASA has prepared 
NAOMS to be handed off to another organization to continue the data collection in 
the future. Two organizations that have been mentioned in this regard are the Air 
Line Pilots Association (ALFA) and the Commercial Aviation Safety Team (CAST). 

I believe that such a hand-off would be unwise, untenable, and unlikely to lead 
to successful continuation of NAOMS data collection. The reason is that within the 
aviation safety community, NASA is uniquely qualified to carry out this work in an 
optimal form, for a series of reasons. 

First, NASA has built up a unique credibility and trust in the aviation safety com- 
munity by running ASRS successfully over the years. No other agency has the trust 
of all interested parties inside and outside of government the way NASA does. This 
trust will enhance the likelihood that pilots, air traffic controllers, flight attendants, 
and mechanics will agree to participate in survey interviews. NASA’s reputation for 
scientific excellence is especially important to allow NAOMS data to earn the trust 
that they deserve. 

Second, NASA has the scientific credibility and third-party objectivity to be able 
to collect data at a distance from those who run airlines, manufacture aircraft, and 
fly on those aircraft. If the data collection were to be run by any interested party, 
their values might be perceived, rightly or wrongly, to have influenced the results 
they obtain and/or distribute. This is a context in which government oversight and 
management of an information collection system run by a private sector contractor 
with considerable expertise is the best way to allow that system to be most effective 
and most helpful to all who can benefit from it. 

Most importantly, I have not heard of any commitment made by ALFA, CAST, 
or any other private sector organization to commit funds to initiate and maintain 
continued NAOMS data collection using the same high-quality methodology that 
NASA developed. The benefits of ASRS data are obvious to all who use that growing 
data set of anecdotes. Considerable added value can and should be created by mak- 
ing long-term commitment through appropriate funding to allow NASA to restart 
NAOMS data collection from pilots, air traffic controllers, flight attendants, and me- 
chanics. 

The Members of this committee fly on commercial airlines, as do huge numbers 
of your constituents, including me. I believe that we all deserve to fly on the safest 
possible system. NASA’s efforts in building and carrying out NAOMS offer the op- 
portunity to significantly enhance our safety by watching carefully what happens in 
real time and documenting risk-elevating events in ways that enable minimization 
of them. As the aviation system grows and changes in the coming years, keeping 
a close eye on its functioning can only increase public confidence in air travel. I 
therefore urge this committee to please take this opportunity to do what I believe 
your constituents would want: to reactive this valuable system under NASA’s roof. 
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Conclusion 

The U.S. Federal Government in general and NASA in particular have a great 
deal to be proud of regarding NAOMS. NAOMS was intended to fill a hole by cre- 
ating an ongoing pipeline of valuable information for the public and for the private 
sector to enhance the welfare of all Americans. It has succeeded in doing so and 
can continue to do so in the future. Thank you for taking this opportunity to con- 
sider assuring that to happen. 
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ating a peer leadership intervention to prevent adolescent substance abuse. 
Paper presented at the American Psychological Association Annual Meeting, 
Montreal, Canada. 

McAlister, A.L., Gordon, N.P., Krosnick, J.A., & Milburn, M.A. (1982). Experimental 
and correlational tests of a theoretical model for smoking prevention. Paper pre- 
sented at the Society for Behavioral Medicine Annual Meeting, Chicago, Illinois. 

Kinder, D.R., Iyengar, S., Krosnick, J.A., & Peters, M.D. (1983). More than meets 
the eye: The impact of television news on evaluations of presidential perform- 
ance. Paper presented at the Midwest Political Science Association Annual 
Meeting, Chicago, Illinois. 

Krosnick, J.A. (1983). The relationship of attitude centrality to attitude stability. 
Paper presented at the American Sociological Association Annual Convention, 
Detroit, Michigan. 
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Alwin, D.F., & Krosnick, J.A. (1984). The measurement of values: A comparison of 
ratings and rankings. Paper presented at the American Association for Public 
Opinion Research Annual Meeting, Delavan, Wisconsin. 

Schuman, H., Ludwig, J., & Krosnick, J.A. (1984). Measuring the salience and im- 
portance of public issues over time. Paper presented at the American Associa- 
tion for Public Opinion Research Annual Meeting, Delavan, Wisconsin. 

Krosnick, J.A. (1984). Attitude extremity, stability, and self-report accuracy: The ef- 
fects of attitude centrality. Paper presented at the American Association for 
Public Opinion Research Annual Meeting, Delavan, Wisconsin. 

Krosnick, J.A. (1984). The influence of consensus information on predictions of one’s 
own behavior. Paper presented at the American Psychological Association An- 
nual Meeting, Toronto, Canada. 

Krosnick, J.A., & Alwin, D.F. (1986). An evaluation of a cognitive theory of response 
order effects in survey measurement. Paper presented at the American Associa- 
tion for Public Opinion Research Annual Meeting, St. Petersburg, Florida. 

Krosnick, J.A. (1986). A new look at question order effects in surveys. Paper pre- 
sented at the Symposium on Cognitive Sciences and Survey Research, Ann 
Arbor, Michigan. 

Krosnick, J.A. (1987). The role of attitude importance in social evaluation: A study 
of policy preferences, presidential candidate evaluations, and voting behavior. 
Paper presented at the Midwest Political Science Association Annual Meeting, 
Chicago, Illinois. 

Krosnick, J.A., Schuman, H., Carnot, C., Berent, M., & Boninger, D. (1987). Attitude 
importance and attitude accessibility. Paper presented at the Midwest Psycho- 
logical Association Annual Meeting, Chicago, Illinois. 

Krosnick, J.A., & Sedikides, C. (1987). Self-monitoring and self-protective biases in 
use of consensus information to predict one’s own behavior. Paper presented at 
the Midwest Psychological Association Annual Meeting, Chicago, Illinois. 

Krosnick, J.A., Stephens, L., Jussim, L.J., & Lynn, A.R. (1987). Sublimhinal priming 
of affect and its cognitive consequences. Paper presented at the Midwest Psy- 
chological Association Annual Meeting, Chicago, Illinois. 

Krosnick, J.A., & Alwin, D.F. (1987). Satisficing: A strategy for dealing with the de- 
mands of survey questions. Paper presented at the American Association for 
Public Opinion Research Annual Meeting, Hershey, Pennsylvania. 

Judd, C.M., & Krosnick, J.A. (1987). The structural bases of consistency among po- 
litical attitudes: The effects of political expertise and attitude importance. Paper 
presented at the American Psychological Association Annual Meeting, New 
York, New York. 

Krosnick, J.A., & Milburn, M.A. (1987). Psychological determinants of political 
opinionation. Paper presented at the American Political Science Association An- 
nual Meeting, Chicago, Illinois. 

Krosnick, J.A. (1987). The role of attitude importance in social evaluation: A study 
of policy preferences, presidential candidate evaluations, and voting behavior. 
Paper presented at the Society for Experimental Social Psychology Annual 
Meeting, Charlottesville, Virginia. 

Krosnick, J.A. (1988). Psychological perspectives on political candidate perception: 
A review of research on the projection hypothesis. Paper presented at the Mid- 
west Political Science Association Annual Meeting, Chicago, Illinois. 

Krosnick, J.A., Boninger, D.S., Berent, M.K., & Carnot, C.G. (1988). The origins of 
attitude importance. Paper presented at the Midwest Psychological Association 
Annual Meeting, Chicago, Illinois. 

Krosnick, J.A., Carnot, C.G., Berent, M.K., & Boninger, D.S. (1988). An exploration 
of the relations among dimensions of attitude strength. Paper presented at the 
Midwest Psychological Association Annual Meeting, Chicago, Illinois. 

Krosnick, J.A., Li, F., & Ashenhurst, J. (1988). Order of information presentation 
and the effect of base-rates on social judgments. Paper presented at the Mid- 
west Psychological Association Annual Meeting, Chicago, Illinois. 

Krosnick, J.A., Berent, M.K., Carnot, C.G., & Boninger, D.S. (1988). Attitude impor- 
tance and recall of attitude relevant information.Paper presented at the Mid- 
west Psychological Association Annual Meeting, Chicago, Illinois. 

Krosnick, J.A., & Carnot, C.G. (1988). A comparison of two theories of the origins 
of political attitude strength. Paper presented at the Midwest Psychological As- 
sociation Annual Meeting, Chicago, Illinois. 
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Krosnick, J.A., & Alwin, D.F. (1988). The stability of political attitudes across the 
life span. Paper presented at the American Association for Public Opinion Re- 
search Annual Meeting, Toronto, Canada. 

Krosnick, J.A., & Carnot, C.G. (1988). Identif 3 dng the foreign affairs attentive pub- 
lic: A comparison of competing theories. Paper presented to the Mershon Center 
Seminar on Foreign Policy Decision Making, The Ohio State University, Colum- 
bus, Ohio. 

Alwin, D.F., & Krosnick, J.A. (1988). The reliability of attitudinal survey data. 
Paper presented at the International Conference on Social Science Methodology, 
Dubrovnik, Yugoslavia. 

Alwin, D.F., & Krosnick, J.A. (1988). Aging, cohort stability, and change in socio- 
political attitudes: Exploring the generational-persistence model. Paper pre- 
sented at the International Society of Political Psychology Annual Meeting, 
Secaucus, New Jersey. 

Krosnick, J.A., & Kinder, D.R. (1988). Altering the foundations of popular support 
for the president through priming: Reagan, the Iran-Contra affair, and the 
American public. Paper presented at the American Political Science Association 
Annual Meeting, Washington, D.C. 

Krosnick, J.A., & Weisberg, H.F. (1988). Liberal/conservative ideological structures 
in the mass public: A study of attitudes toward politicians and social groups. 
Paper presented at the American Political Science Association Annual Meeting, 
Washington, D.C. 

Krosnick, J.A. (1988). Government policy and citizen passion: A study of issue 
publics in contemporary America. Paper presented at the Shambaugh Con- 
ference on Communication, Cognition, Political Judgment, and Affect, Iowa 
City, Iowa. 

Berent, M.K., Krosnick, J.A., & Boninger, D.S. (1989). Attitude importance and the 
valanced recall of relevant information. Paper presented at the Midwest Psycho- 
logical Association Annual Meeting, Chicago, Illinois. 

Betz, A., & Krosnick, J.A. (1989). Can people detect the affective tone of sublimi- 
nally presented stimuli? Paper presented at the Midwest Psychological Associa- 
tion Annual Meeting, Chicago, Illinois. 

Krosnick, J.A., & Berent, M.K. (1989). Age-related changes in peer and parental in- 
fluence on heavy television viewing among children and adolescents. Paper pre- 
sented at the Midwest Psychological Association Annual Meeting, Chicago, Illi- 
nois. 

Alwin, D.F., & Krosnick, J.A. (1989). The reliability of attitudinal survey data. 
Paper presented at the American Association for Public Opinion Research An- 
nual Meeting, St. Petersburg, Florida. 

Krosnick, J.A. (1989). The implications of social psychological findings on compliance 
for recruiting survey respondents. Paper presented at the American Association 
for Public Opinion Research Annual Meeting, St. Petersburg, Florida. 

Telhami, S., & Krosnick, J.A. (1989). Public attitudes and American policy toward 
the Arab-Israeli conflict. Paper presented at the International Society of Polit- 
ical Psychology Annual Meeting, Israel. 

Krosnick, J.A., & Alwin, D.F. (1989). Symbolic versus non-sjmibolic political atti- 
tudes: Is there a distinction? Paper presented at the American Political Science 
Association Annual Meeting, Atlanta, Georgia. 

Krosnick, J.A. (1989). The impact of cognitive sophistication and attitude impor- 
tance on response order effects and question order effects. Paper presented at 
the conference entitled Order effects in social and psychological research. Nags 
Head Conference Center, Kill Devil Hills, North Carolina. 

Krosnick, J.A. (1990). The impact of satisficing on survey data quality. Paper pre- 
sented at the Annual Research Conference of the Bureau of the Census, U.S. 
Department of Commerce, Washington, D.C. 

Krosnick, J.A. (1990). New perspectives on survey questionnaire construction: Les- 
sons from the cognitive revolution. Invited presentation at the 1990 Technical 
Conference of the United States General Accounting Office, College Park, Mary- 
land. 

Krosnick, J.A. (1990). Americans’ perceptions of presidential candidates: A test of 
the projection hypothesis. Paper presented at the Midwest Political Science As- 
sociation Annual Meeting, Chicago, Illinois. 
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Krosnick, J.A., & Berent, M.K. (1990). The impact of verbal labeling of response al- 
ternatives and branching on attitude measurement reliability in surveys. Paper 
presented at the American Association for Public Opinion Research Annual 
Meeting, Lancaster, Pennsylvania. 

Krosnick, J.A., & Alwin, D.F. (1990). The stability of political preferences: Compari- 
sons of symbolic and non-symbolic attitudes. Paper presented at the Inter- 
national Society of Political Psychology Annual Meeting, Washington, D.C. 

Krosnick, J.A. (1990). Confounding of attitude objects with attitude measurement 
techniques in studies of political attitude stability. Paper presented at the Sum- 
mer Institute in Survey Research Techniques, University of Michigan. 

Fabrigar, L.R., & Krosnick, J.A. (1991). The effect of question order and attitude im- 
portance on the false consensus effect. Paper presented at the Midwestern Psy- 
chological Association Annual Meeting, Chicago, Illinois. 

Berent, M.K., & Krosnick, J.A. (1991). Attitude measurement reliability: The impact 
of verbal labeling of response alternatives and branching. Paper presented at 
the Midwestern Psychological Association Annual Meeting, Chicago, Illinois. 

Lehman, D.R., Krosnick, J.A., West, R.L., & Li, F. (1991). The focus of judgment 
effect: A question wording effect due to hypothesis confirmation bias. Paper pre- 
sented at the American Association for Public Opinion Research Annual Meet- 
ing, Phoenix, Arizona. 

Krosnick, J.A., Boninger, D.S., Chuang, Y.C., & Carnot, C.G. (1991). Attitude 
strength: One construct or many related constructs? Paper presented at the 
Nags Head Conference on Attitude Strength, Nags Head, North Carolina. 

Krosnick, J.A. (1991). Research on attitude importance: A summary and integration. 
Paper presented at the Nags Head Conference on Attitude Strength, Nags 
Head, North Carolina. 

Krosnick, J.A., & Berent, M.K. (1991). Memory for political information: The impact 
of attitude importance on selective exposure, selective elaboration, and selective 
recall. Paper presented at the Society for Experimental Social Psychology An- 
nual Meeting, Columbus, Ohio. 

Krosnick, J.A., & Brannon, L.A. (1992). The impact of war on the ingredients of 
presidential evaluations: George Bush and the Gulf conflict. Paper presented at 
the Conference on the Political Consequences of War, The Brookings Institution, 
Washington, D.C. 

Berent, M.K., & Krosnick, J.A. (1992). The relation between attitude importance 
and knowledge structure. Paper presented at the Midwest Political Science As- 
sociation Annual Meeting, Chicago, Illinois. 

Smith, W.R., Culpepper, I.J., & Krosnick, J.A. (1992). The impact of question order 
on cognitive effort in survey responding. Paper presented at the Sixth National 
Conference on Undergraduate Research, University of Minnesota, Minneapolis, 
Minnesota. 

Krosnick, J.A., & Brannon, L.A. (1992). The impact of war on the ingredients of 
presidential evaluations: George Bush and the Gulf conflict. Paper presented at 
the American Association for Public Opinion Research Annual Meeting, St. Pe- 
tersburg, Florida. 

Narayan, S.S., & Krosnick, J.A. (1992). Response effects in surveys as a function 
of cognitive sophistication. Paper presented at the Midwestern Psychological As- 
sociation Annual Meeting, Chicago, Illinois. 

Boninger, D.S., Krosnick, J.A., & Berent, M.K. (1992). Imagination, perceived likeli- 
hood, and self-interest: A path toward attitude importance. Paper presented at 
the Midwestern Psychological Association Annual Meeting, Chicago, Illinois. 

Culpepper, I.J., Smith, W., & Krosnick, J.A. (1992). The impact of question order 
on satisficing in attitude surveys. Paper presented at the Midwestern Psycho- 
logical Association Annual Meeting, Chicago, Illinois. 

Berent, M.K., & Krosnick, J.A. (1992). Attitude importance, information accessi- 
bility, and attitude-relevant judgments. Paper presented at the Midwestern Psy- 
chological Association Annual Meeting, Chicago, Illinois. 

Krosnick, J.A., & Brannon, L.A. (1992). The impact of war on the ingredients of 
presidential evaluations: George Bush and the Gulf conflict. Paper presented at 
the International Society of Political Psychology Annual Meeting, San Fran- 
cisco, California. 
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Rahn, W.M., Krosnick, J.A., & Breuning, M. (1992). Rationalization and derivation 
processes in political candidate evaluation. Paper presented at the American Po- 
litical Science Association Annual Meeting, Chicago, Illinois. 

Krosnick, J.A., & Brannon, L.A. (1992). Effects of knowledge, interest, and exposure 
on news media priming effects: Surprising results from multivariate analysis. 
Paper presented at the Society for Experimental Social Psychology Annual 
Meeting, San Antonio, Texas. 

Berent, M.K., & Krosnick, J.A. (1993). Attitude importance and selective exposure 
to attitude-relevant information. Paper presented at the Midwestern Psycho- 
logical Association Annual Meeting, Chicago, Illinois. 

Fabrigar, L.R., & Krosnick, J.A. (1993). The impact of personal and national impor- 
tance judgments on political attitudes and behavior. Paper presented at the 
Midwestern Psychological Association Annual Meeting, Chicago, Illinois. 

Miller, J.M., & Krosnick, J.A. (1993). The effects of candidate ballot order on elec- 
tion outcomes. Paper presented at the Midwestern Psychological Association 
Annual Meeting, Chicago, Illinois. 

Narayan, S.S., & Krosnick, J.A. (1993). Questionnaire and respondents characteris- 
tics that cause satisficing in attitude surveys. Paper presented at the Mid- 
western Psychological Association Annual Meeting, Chicago, Illinois. 

Narayan, S.S., & Krosnick, J.A. (1993). Response effects in surveys as a function 
of cognitive sophistication. Paper presented at the American Psychological Soci- 
ety Annual Meeting, Chicago, Illinois. 

Smith, W.R., & Krosnick, J.A. (1993). Need for cognition, prior thought, and 
satisficing in attitude surveys. Paper presented at the Midwestern Psychological 
Association Annual Meeting, Chicago, Illinois. 

Smith, W.R., & Krosnick, J.A. (1993). Cognitive and motivational determinants of 
satisficing in surveys. Paper presented at the American Psychological Society 
Annual Meeting, Chicago, Illinois. 

Berent, M.K., & Krosnick, J.A. (1994). Attitude importance and selective exposure 
to attitude-relevant information. Paper presented at the Midwest Political 
Science Association Annual Meeting, Chicago, Illinois. 

Fabrigar, L.R., & Krosnick, J.A. (1994). The impact of attitude importance on con- 
sistency among attitudes. Paper presented at the Midwestern Psychological As- 
sociation Annual Meeting, Chicago, Illinois. 

Krosnick, J.A. (1994). Survey methods and survey results: Overturing conventional 
wisdom. Paper presented to the American Marketing Association, Columbus 
Chapter. 

Krosnick, J.A., & Fabrigar, L.R. (1994). Attitude recall questions: Do they work? 
Paper presented at the American Association for Public Opinion Research An- 
nual Meeting, Danvers, Massachusetts. 

Miller, J.M., & Krosnick, J.A. (1994). Does accessibility mediate agenda-setting and 
priming? Paper presented at the Midwestern Psychological Association Annual 
Meeting, Chicago, Illinois. 

Smith, W.R., & Krosnick, J.A. (1994). Sources of non-differentiation and mental 
coin-flipping in surveys: Tests of satisficing hypotheses. Paper presented at the 
American Association for Public Opinion Research Annual Meeting, Danvers, 
Massachusetts. 

Visser, P.S., & Krosnick, J.A. (1994). Mail surveys for election forecasting? An eval- 
uation of the Columbus Dispatch Poll. Paper presented at the Midwestern Psy- 
chological Association Annual Meeting, Chicago, Illinois. 

Visser, P.S., Krosnick, J.A., & Curtin, M. (1994). Mail surveys for election fore- 
casting? Paper presented at the American Association for Public Opinion Re- 
search Annual Meeting, Danvers, Massachusetts. 

Krosnick, J.A., & Brannon, L.A. (1995). News media priming and the 1992 U.S. 
presidential election. Paper presented at the American Political Science Associa- 
tion Annual Meeting, Chicago, Illinois. 

Krosnick, J.A., & Cornet, P.J. (1995). Attitude importance and attitude change re- 
visited: Shifts in attitude stability and measurement reliability across a presi- 
dential election campaign. Paper presented at the American Psychological Soci- 
ety Annual Meeting, New York, New York. 

Krosnick, J.A., & Fabrigar, L.R. (1995). Designing rating scales for effective meas- 
urement in surveys. Invited address at the International Conference on Survey 
Measurement and Process Quality, Bristol, England. 
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Krosnick, J.A., Narayan, S.S., & Smith, W.R. (1995). The causes of survey 
satisficing: Cognitive skills and motivational factors. Paper presented at the 
Midwest Association for Public Opinion Research, Chicago, Illinois. 

Miller, J.M., Fahrigar, L.R., & Krosnick, J.A. (1995). Contrasting attitude impor- 
tance and collective issue importance: Attitude properties and consequences. 
Paper presented at the Midwestern Psychological Association Annual Meeting, 
Chicago, Illinois. 

Miller, J.M., & Krosnick, J.A. (1995). Ballot order effects on election outcomes. 
Paper presented at the Midwest Political Science Association Annual Meeting, 
Chicago, Illinois. 

Miller, J.M., & Krosnick, J.A. (1995). Mediators and moderators of news media 
priming: It ain’t accessibility, folks. Paper presented at the International Soci- 
ety of Political Psychology Annual Meeting, Washington, D.C. 

Narayan, S.S., & Krosnick, J.A. (1995). Education moderates response effects in sur- 
veys. Paper presented at the American Association for Public Opinion Research 
Annual Meeting, Ft. Lauderdale, Florida. 

Smith, W.R., & Krosnick, J.A. (1995). Mental coin-flipping and non-differentiation 
in surveys: Tests of satisficing hypotheses. Invited address at the Midwestern 
Psychological Association Annual Meeting, Chicago, Illinois. 

Visser, P.S., & Krosnick, J.A. (1995). The relation between age and susceptibility 
to attitude change: A new approach to an old question. Paper presented at the 
Midwestern Psychological Association Annual Meeting, Chicago, Illinois. 

Visser, P.S., & Krosnick, J.A. (1995). Mail surveys win again: Some explanations for 
the superior accuracy of the Columbus Dispatch poll. Paper presented at the 
American Association for Public Opinion Research Annual Meeting, Ft. Lauder- 
dale, Florida. 

Ankerbrand, A.L., Krosnick, J.A., Cacioppo, J.T., & Visser, P.S. (1996). Candidate 
assessments and evaluative space. Paper presented at the Midwestern Psycho- 
logical Association Annual Meeting, Chicago, Illinois. 

Bizer, G.Y., & Krosnick, J.A. (1996). Attitude accessibility and importance revisited. 
Paper presented at the Midwestern Psychological Association Annual Meeting, 
Chicago, Illinois. 

Krosnick, J.A. (1996). Linking survey question structure to data quality: The impact 
of no-opinion options. Paper presented at the conference on “Quality Criteria in 
Survey Research,” sponsored by the World Association for Public Opinion Re- 
search, Cadenabbia, Italy. 

Krosnick, J.A., & Brannon, L.A. (1996). News media priming during the 1992 U.S. 
presidential election campaign. Paper presented at the International Society of 
Political Psychology Annual Meeting, Vancouver, British Columbia. 

Miller, J.M., Fabrigar, L.R., & Krosnick, J.A. (1996). The roles of personal impor- 
tance and national importance in motivating issue public membership. Paper 
presented at the Midwest Political Science Association Annual Meeting, Chi- 
cago, Illinois. 

Miller, J.M., & Krosnick, J.A. (1996). Can issue public membership be triggered by 
the threat of a policy change? Paper presented at the International Society of 
Political Psychology Annual Meeting, Vancouver, British Columbia. 

Krosnick, J.A., & Visser, P.S. (1996). Changes in political attitude strength through 
the life cycle. Paper presented at the Society for Experimental Social Psychology 
Annual Meeting, Sturbridge, Massachusetts. 

Miller, J.M., & Krosnick, J.A. (1997). The impact of policy change threat on issue 
public membership. Paper presented at the Midwest Political Science Associa- 
tion Annual Meeting, Chicago, Illinois. 

Ankerbrand, A.L., Krosnick, J.A., Cacioppo, J.T., Visser, P.S., & Gardner, W. (1997). 
Attitudes toward political candidates predict voter turnout. Paper presented at 
the Midwestern Psychological Association Annual Meeting, Chicago, Illinois. 

Ankerbrand, A.L., & Krosnick, J.A. (1997). Response order effects in dichotomous 
questions: A social desirability explanation. Paper presented at the American 
Psychological Society Annual Meeting, Washington, DC. 

Krosnick, J.A. (1997). Miraculous accuracy in political surveys: The keys to success. 
Presentation in the Federation of Behavioral, Psychological, and Cognitive 
Sciences Seminar on Science and Public Policy, Library of Congress, Wash- 
ington, D.C. 
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Krosnick, J.A. (1997). Non-attitudes and no-opinion filters. Paper presented at the 
Conference on no opinion, instability, and change in public opinion research. 
University of Amsterdam, the Netherlands. 

Krosnick, J.A. (1997). Attitude strength. Paper presented at the Conference on no 
opinion, instability, and change in public opinion research. University of Am- 
sterdam, the Netherlands. 

Bizer, G.Y., & Krosnick, J.A. (1998). The relation between attitude importance and 
attitude accessibility. Paper presented at the Midwestern Psychological Associa- 
tion Annual Meeting, Chicago, Illinois. 

Holbrook, A., Krosnick, J.A., Carson, R.T., & Mitchell, R.C. (1998). Violating con- 
versational conventions disrupts cognitive processing of survey questions. Paper 
presented at the American Association for Public Opinion Research Annual 
Meeting, St. Louis, Missouri. 

Krosnick, J.A. (1998). Applying stated preference methods to assessing the value of 
public goods. Paper presented at the National Oceanic and Atmospheric Admin- 
istration Application of Stated Preference Methods to Resource Compensation 
Workshop, Washington, DC. 

Krosnick, J.A. (1998). Implications of psychological research on justice and com- 
pensation for handling of natural resource damage cases. Paper presented at 
the National Oceanic and Atmospheric Administration Application of Stated 
Preference Methods to Resource Compensation Workshop, Washington, DC. 

Krosnick, J.A. (1998). Acquiescence: How a standard practice in many survey orga- 
nizations compromises data quality. Paper presented at the conference on 
“Quality Criteria in Survey Research,” sponsored by the World Association for 
Public Opinion Research, Cadenabbia, Italy. 

Krosnick, J.A., Lacy, D., & Lowe, L. (1998). When is environmental damage Ameri- 
cans’ most important problem? A test of agenda-setting vs. the issue-attention 
cycle. Paper presented at the International Society of Political Psychology An- 
nual Meeting, Montreal, Quebec, Canada. 

Visser, P.S., Krosnick, J.A., Marquette, J., & Curtin, M. (1998). Improving election 
forecasting: Allocation of undecided respondents, identification of likely voters, 
and response order effects. Paper presented at the American Association for 
Public Opinion Research Annual Meeting, St. Louis, Missouri. 

Krosnick, J.A. (1998). The impact of science on public opinion: How people judge the 
national seriousness of global warming and form policy preferences. Paper pre- 
sented at the American Political Science Association Annual Meeting, Boston, 
Massachusetts. 

Krosnick, J.A. (1998). Response choice order and attitude reports: New evidence on 
conversational conventions and information processing biases in voting and in 
election forecasting polls. Paper presented at the Society of Experimental Social 
Psychology Annual Meeting, Lexington, Kentucky. 

Krosnick, J.A. (1998). The impact of the Fall 1997 debate about global warming on 
American public opinion. Paper presented at Resources for the Future, Wash- 
ington, D.(i. 

Krosnick, J.A. (1998). What the American public believes about global warming: Re- 
sults of a national longitudinal survey study. Paper presented at the Amoco 
Public and Government Affairs and Government Relations Meeting, Woodruff, 
Wisconsin. 

Krosnick, J.A. (1998). What the American public believes about global warming: Re- 
sults of a national longitudinal survey study. Paper presented in the Second An- 
nual Carnegie Lectures on Global Environmental Change, Carnegie Museum of 
Natural History, Pittsburgh, Pennsylvania. 

Green, M.C., & Krosnick, J.A. (1999). Survey satisficing: Telephone interviewing in- 
creases non-differentiation and no opinion responses. Paper presented at the 
Midwestern Psychological Association Annual Meeting, Chicago, Illinois. 

Green, M.C., & Krosnick, J.A. (1999). Comparing telephone and face-to-face inter- 
viewing in terms of data quality: The 1982 National Election Studies Method 
Comparison Project. Paper presented at the Seventh Annual Conference on 
Health Survey Research Methods, Williamsburg, Virginia. 

Holbrook, A.L., Krosnick, J.A., Carson, R.T., & Mitchell, R.C. (1999). Violating con- 
versational conventions disrupts cognitive processing of attitude questions. 
Paper presented at the American Association for Public Opinion Research An- 
nual Meeting, St. Petersburg, Florida. 
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Krosnick, J.A. (1999). What happens when survey respondents don’t try very hard? 
The notion of survey satisficing. Paper presented at the National Center for So- 
cial Research, London, United Kingdom. 

Krosnick, J.A. (1999). Satisficing: A single explanation for a wide range of findings 
in the questionnaire design literature. Paper presented at Linking the Path: A 
Conference for Analysts, Researchers, and Consultants, sponsored by the Gallup 
Organization, Lincoln, Nebraska. 

Krosnick, J.A. (1999). Methodology for the NAOMS Survey. Presentation at the 
Workshop on the Concept of the National Aviation Operations Monitoring Sys- 
tem (NAOMS), Sponsored by the National Aeronautics and Space Administra- 
tion, Alexandria, Virginia. 

Krosnick, J.A. (1999). Refining measurement of public values for policy-making: A 
test of contingent valuation procedures. Paper presented at the American Polit- 
ical Science Association Annual Meeting, Atlanta, Georgia. 

Krosnick, J.A. (1999). The threat of satisficing in surveys: The shortcuts respond- 
ents take in answering questions. Paper presented at the National Center for 
Social Research Survey Methods Seminar on Survey Data Quality, London, 
England. 

Krosnick, J.A. (1999). Optimizing questionnaire design: How to maximise data qual- 
ity. Paper presented at the National Center for Social Research Survey Methods 
Seminar on Survey Data Quality, London, England. 

Krosnick, J.A. (1999). The causes and consequences of no-opinion responses in sur- 
veys. Paper presented at the International Conference on Survey Nonresponse, 
Portland, Oregon. 

Miller, J.M., & Krosnick, J.A. (1999). The impact of threats and opportunities on 
political participation. Paper presented at the Midwest Political Science Associa- 
tion Annual Meeting, Chicago, Illinois. 

O’Muircheartaigh, C., Krosnick, J.A., & Helic, A. (1999). Middle alternatives, acqui- 
escence, and the quality of questionnaire data. Paper presented at the American 
Association for Public Opinion Research Annual Meeting, St. Petersburg, Flor- 
ida. 

Bizer, G.Y., & Krosnick, J.A. (2000). The importance and accessibility of attitudes: 
Helping explain the structure of strength-related attitude attributes. Paper pre- 
sented at the Midwestern Psychological Association Annual Meeting, Chicago, 
Illinois. 

Holbrook, A.L., Krosnick, J.A., Visser, P.S., Gardner, W.L., & Cacioppo, J.T. (2000). 
The formation of attitudes toward presidential candidates and political parties: 
An asymmetric nonlinear process. Paper presented at the American Psycho- 
logical Society Annual Meeting, Miami, Florida. 

Holbrook, A.L., Krosnick, J.A., Visser, P.S., Gardner, W.L., & Cacioppo, J.T. (2000). 
The formation of attitudes toward presidential candidates and political parties: 
An as 3 mimetric, nonlinear, interactive process. Paper presented at the American 
Political Science Association Annual Meeting, Washington, D.C. 

Krosnick, J.A. (2000). Peering into the future of thinking and answering: A psycho- 
logical perspective on internet survey respondents. Paper presented at Survey 
Research: Past, Present, and Internet, the 2000 Nebraska Symposium on Survey 
Research, University of Nebraska, Lincoln, Nebraska. 

Krosnick, J.A. (2000). The present and future of research on survey non-responses: 
Reflections on Portland ’99 and beyond. Roundtable presentation at the Amer- 
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Study of Law and Politics and the Jesse M. Unruh Institute of Politics, Univer- 
sity of Southern California, Los Angeles, California. 
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Chang, L., & Krosnick, J.A. (2001). The accuracy of self-reports: Comparisons of an 
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Research Annual Meeting, Montreal, Canada. 
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Krosnick, J.A. (2001). Americans’ perceptions of the health risks of cigarette smok- 
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ference “Survey Research on Household Expectations and Preferences,” Insti- 
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ington. 



84 
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Silver, M.D., & Krosnick, J.A. (2001). Optimizing survey measurement accuracy by 
matching question design to respondent memory organization. Paper presented 
at the Federal Committee on Statistical Methodology Research Conference, Ar- 
lington, Virginia. 
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Chang, L., & Krosnick, J.A. (2002). RDD telephone vs. Internet survey methodology 
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microscope: Do classic experiments replicate when participants are representa- 
tive of the general public rather than convenience samples of college students? 
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Chang, L., & Krosnick, J.A. (2002). RDD telephone vs. Internet survey methodology 
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Visser, P.S., & Krosnick, J.A. (2003). Attitude strength: New insights from a life- 
course development perspective. Paper presented at the Society for Personality 
and Social Psychology Annual Meeting, Los Angeles, California. 
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Holbrook, A.L., & Krosnick,, J.A. (2003). Meta-psychological and operative measures 
of psychological constructs: The same or different? Paper presented at the Mid- 
western Psychological Association Annual Meeting, Chicago, Illinois. 

Krosnick, J.A., Visser, P.S., & Holbrook, A.L. (2003). Social psychology under the 
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Krosnick, J.A., Visser, P.S., & Holbrook, A.L. (2003). How to conceptualize attitude 
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Chang, L., & Krosnick, J.A. (2003). Comparing data quality in telephone and inter- 
net surveys: Results of lab and field experiments. Invited paper presented at 
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Pfent, A., & Krosnick, J.A. (2003). Post-decisional dissonance reduction by a new 
method: Rationalization of political candidate choices illuminates the basic dy- 
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Krosnick, J.A., & Fabrigar, L.R. (2003). “Don’t know” and “no opinion” responses: 
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Large Social Surveys, University of Basel, Basel, Switzerland. 
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conference entitled “Integrating Message Effects and Behavior Change Theories 
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Krosnick, J.A. (2003). Survey methodology — scientific basis. Presentation at the Na- 
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Krosnick, J.A. (2003). Survey methodology — NAOMS design decisions. Presentation 
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ing #1, Seattle, Washington. 
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Krosnick, J.A. (2004). Survey methodology — NAOMS design decisions. Presentation 
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at the International Symposium in Honour of Paul Lazarsfeld, Katholieke 
Universiteit Leuven (Belgium). 

Krosnick, J.A., Shook, N., & Thomas, R.K. (2004). Public opinion change in the 
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Holbrook, A.L., & Krosnick, J.A. (2004). Vote over-reporting: A test of the social de- 
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Chang, L., & Krosnick, J.A. (2004). Assessing the accuracy of event rate estimates 
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Chang, L., & Krosnick, J.A. (2004). Measuring the frequency of regular behaviors: 
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Krosnick, J.A. (2004). What do Americans want government to do about global 
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shop on Global Warming: The Psychology of Long Term Risk,” Cooperative In- 
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Krosnick, J.A., & Malhotra, N. (2004). The causes of vote choice in the 2004 Amer- 
ican Presidential Election: Insights from the 2004 YouGov surveys. Paper pre- 
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sented at the conference “The 2004 American Presidential Election: Voter Deci- 
sion-Making in a Complex World,” Stanford University, Stanford, California. 

Krosnick, J.A., Visser, P.S., & Holbrook, A.L. (2004). The impact of social psycho- 
logical manipulations embedded in surveys on special populations. Paper pre- 
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Research Annual Meeting, San Francisco, California. 

Krosnick, J.A. (2005). The future of the American National Election Studies. Round- 
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Krosnick, J.A., & Rivers, D. (2005). Web survey methodologies: A comparison of sur- 
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Krosnick, J.A. (2005). Thought piece on survey participation. Paper presented at the 
conference entitled “New Approaches to Understanding Participation in Sur- 
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DC. 
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Malhotra, N., & Krosnick, J.A. (2005). The determinants of vote choice in the 2004 
U.S. Presidential Election. Paper presented at the American Political Science 
Association Annual Meeting, Washington, DC. 

Krosnick, J.A. (2005). Effects of survey data collection mode on response quality: Im- 
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conference “Mixed Mode Data Collection in Comparative Social Surveys,” City 
University, London, United Kingdom. 

Krosnick, J.A., & Malhotra, N. (2006). The impact of presidential job performance 
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Rabinowitz, J.L. & Krosnick, J.A. (2006). Investigating the discriminant validity of 
symbolic racism. Paper presented at the annual meeting of the Society for Per- 
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Krosnick, J.A. (2006). An evaluation framework: Total survey error in research prac- 
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Insights, Microsoft, Redmond, Washington. 

Krosnick, J.A. (2006). The distinguishing characteristics of frequent survey partici- 
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Krosnick, J.A. (2006). An overview of the mission of the American National Election 
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Krosnick, J.A. (2006). The use of the internet in valuation surveys. Presentation at 
the workshop “Morbidity and Mortality: How Do We Value the Risk of Illness 
and Death?”, sponsored by the U.S. Environmental Protection Agency, the Na- 
tional Center for Environmental Research, and the National Council on Eco- 
nomic Education, Washington, DC. 

Krosnick, J.A. (2006). What the American public thinks about climate change: Find- 
ings from a new Stanford/ABC/Time Magazine Survey. Presentation at the 
“California Climate Change Policy Workshop,” sponsored by the Woods Institute 
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Holbrook, A.L., & Krosnick, J.A. (2006). Vote over-reporting: A test of the social de- 
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Bannon, B., Krosnick, J.A., & Brannon, L. (2006). News media priming: Derivation 
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Krosnick, J.A. (2006). Overview of the American National Election Studies: Lessons 
learned about the causes of voter turnout and candidate choice. Paper presented 
at the conference “The Psychology of Voting and Election Campaigns,” Social 
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Krosnick, J.A. (2006). What Americans really think about climate change. Presen- 
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brary, Orinda, California. 

Krosnick, J.A. (2006). The impact of survey mode and the merging of face-to-face 
recruitment with Internet data collection. Paper presented at the 2006 Federal 
Committee on Statistical Methodology Statistical Policy Seminar, “Keeping Cur- 
rent: What We Know — What We Need to Learn.” Washington, DC. 

Krosnick, J.A. (2006). Comparisons of the accuracy of information obtained by face- 
to-face, telephone, Internet, and paper and pencil data collection. Paper pre- 
sented at the Pacific Chapter of the American Association for Public Opinion 
Research Annual Meeting, San Francisco, California. 

Bizer, G.Y., Krosnick, J.A., Holbrook, A.L., Wheeler, S.C., Rucker, D.D., & Petty, 
R.E. (2007). The impact of personality on political beliefs, attitudes, and behav- 
ior: Need for cognition and need to evaluate. Paper presented at the Society for 
Personality and Social Psychology Annual Meeting, Memphis, Tennessee. 

Sargent, M.J., Rabinowitz, J., Shull, A., & Krosnick, J.A. (2007). Support for govern- 
ment efforts to promote racial equality: Effects of antigroup affect and percep- 
tions of value violation. Paper presented at the Society for Personality and So- 
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Krosnick, J.A. (2007). Americans’ beliefs about global climate change: New national 
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ment of Science Annual Meeting, San Francisco, California. 

Krosnick, J.A. (2007). Comparisons of survey modes and a new hybrid. Paper pre- 
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Lupia, A., & Krosnick, J.A. (2007). Remaking the American National Election Stud- 
ies. Paper presented at the National Conference of Black Political Scientists, 
Burlingame, California. 

Krosnick, J.A. (2007). What Americans really think about climate change: Attitude 
formation and change in response to a raging scientific controversy. Presen- 
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Krosnick, J.A., & Smith, T. (2007). Proposing questionnaire design experiments for 
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Krosnick, J.A., Malhotra, N., & Miller, L. (2007). Survey mode in the 21st Century: 
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Studies: The Wivenhoe House Conference.” University of Essex, Colchester, UK. 

Pasek, J., & Krosnick, J.A. (2007). Trends over time in America: Probability/tele- 
phone vs. non-probability/internet. Paper presented at the conference entitled 
“Cyberinfrastructure and National Election Studies: The Wivenhoe House Con- 
ference.” University of Essex, Colchester, UK. 
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Carolina, Department of Political Science; University of Chicago, Political Psy- 
chology Workshop, Departments of Psychology and Political Science; Pitzer Col- 
lege, Department of Political Science. 

2003 — University of Illinois at Chicago, College of Urban Planning and Public Af- 
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Dostart Clapp Gordon & Coveney LLP, San Diego, California. 
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United States Government Accountability Office, Washington, DC. 
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Monitor Company, Cambridge, Massachusetts. 

American Association for Public Opinion Research Annual Meeting, St. Louis, Mis- 
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Chairman Gordon. Thank you, Doctor, and Captain McVenes, 
you are recognized. 

STATEMENT OF CAPTAIN TERRY L. MCVENES, EXECUTIVE AIR 

SAFETY CHAIRMAN, AIR LINE PILOTS ASSOCIATION, INTER- 
NATIONAL 

Captain McVenes. Mr. Chairman, Mr. Hall, Members of the 
Committee, good afternoon, and thank you for the opportunity to 
outline the Air Line Pilots Association’s views on aviation safety 
and the role that we play in protecting the traveling public. 

ALPA is the world’s largest pilot union. We represent more than 
60,000 pilots at 42 airlines in the United States and Canada. ALPA 
was founded in 1931, and for more than 76 years now ALPA has 
had a tremendous impact on improving aviation safety. Today 
ALPA continues to be the world’s leading aviation safety advocate, 
protecting the safety interests of our passengers, our fellow crew 
members, and cargo around the world. 

Over the past 10 years the U.S. aviation industry has seen a 65 
percent decrease in the accident rate, and as a result, the U.S. 
safety record is the envy of the rest of the world. Much of our suc- 
cess is due to the collaborative approach that has taken place 
among airline managements, labor, and the FAA in voluntary col- 
lection and analysis of de-identified safety-related data. By ana- 
lyzing recorded data that is obtained during routine flight oper- 
ations and receiving written reports from the front-line employees 
in a confidential and non-punitive environment, we can not only 
see what is happening out there but also why it is happening. 

Today these stand-alone programs at individual airlines are 
reaching their maturity, and that is a reflection of the dynamic na- 
ture of any data collection effort. It has to adapt to changes in the 
environment, and in this case, the changes in the aviation indus- 
try. 

As safety professionals continue to see value in these programs 
and work with them in more detail, it has become clear that even 
more can be learned by sharing safety information among various 
stakeholders in the industry. The FAA and the airline industry, in- 
cluding ALPA, continue to work together on developing a formal- 
ized process in which safety information can be accessed through 
secure networks under mutually-agreeable rules of engagement. 

ALPA has been working closely with the FAA, NASA, and the 
airlines to develop a process that will make the safety information 
available to decision-makers to help them in their efforts to man- 
age risks. This process is also invaluable in the sharing of accident 
and incident prevention strategies across the entire industry. 
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Again, though, I would point out that as time goes on, the indus- 
try continues to refine our processes for maximizing the safety ben- 
efits that the traveling public receives from collecting data while at 
the same time protecting those employees and the airlines that 
bring the data to the table in the first place. 

NASA, especially through the Aviation Safety Reporting System 
or ASRS, has always been an important player in aviation safety. 
Its human factors research in particular has provided great value 
to our industry. The NAOMS survey was part of the early effort to 
provide more information to help all of us improve aviation safety. 
And this first survey was a test of the process and methodology, 
and we understand that the data extracted from this survey were 
summarized, and those summaries were shared with government 
and industry. 

But as in any first test the data didn’t correlate very well with 
data from other sources, possibly due to the mix of general aviation 
and airline operations. The aviation community had plans to fur- 
ther analyze those discrepancies and determine if the data was re- 
liable, but the funding for NAOMS ran out, and that is when ALFA 
stepped in to help keep that project alive as part of our involve- 
ment with the Commercial Aviation Safety Team or CAST. And 
while we have been working with CAST to modify that survey, we 
did not receive any collected data from NASA. 

So what should we do with the data now? Well, there are several 
solutions that are available. We have heard some of them this 
afternoon. The one that makes a lot of sense is to provide NASA 
with the necessary resources so it can complete its peer review of 
the data, then analyze that data, while at the same time maintain 
the confidentiality and protective provisions that apply to volun- 
tarily supplied safety information. 

Other solutions may also exist, but regardless of the solution, it 
is important to keep in mind that raw data distributed without ap- 
propriate analysis and scrutiny to ensure its validity can lead to 
unintended consequences. Incomplete or inaccurate conclusions can 
be reached if the collection method is flawed or if people looking 
at the data aren’t familiar with aviation or the context of how that 
information was provided. 

No one knows and understands the data better than the stake- 
holders that provided the data in the first place. That is why it is 
so important that those stakeholders work closely with the analysts 
of the data, and this will ensure accurate and meaningful conclu- 
sions can be reached. Just as importantly, if raw data is simply 
dumped onto the general public without the quality controls I have 
mentioned, it would undermine the confidence that pilots and the 
airline community that had voluntarily and confidentially supplied 
data and other sources. We have to make sure that that confiden- 
tiality remains secure. 

Now, as an airline captain, one who represents the safety inter- 
ests of 60,000 other airline pilots, I am concerned that this could 
very well erode the very programs that have driven the excellent 
safety record of airline travel that the public has come to rely on. 

Thank you, and again, for the opportunity to testify today, and 
I will be pleased to address any questions you may have. 

[The prepared statement of Captain McVenes follows:] 
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Prepared Statement of Captain Terry L. McVenes 

Good afternoon and thank you for the opportunity to outline the Air Line Pilots 
Association’s views on aviation safety and the role we play in protecting the trav- 
eling public. ALPA is the world’s largest pilot union, representing more than 60,000 
pilots who fly for 42 airlines in the U.S. and Canada. ALPA was founded in 1931, 
and for more than 76 years, ALPA has had a tremendous impact on improving avia- 
tion safety. Today, ALPA continues to be the world’s leading aviation safety advo- 
cate, protecting the safety interests of our passengers, fellow crew members, and 
cargo around the world. 

Over the past 10 years, the U.S. aviation industry has seen a 65 percent decrease 
in the accident rate, and as a result, the U.S. safety record is the envy of the rest 
of the world. Much of our success is due to the collaborative approach that has 
taken place among airline managements, labor, and the FAA in the voluntary collec- 
tion and analysis of de-identified safety related data. By analyzing recorded data ob- 
tained during routine flight operations and receiving written reports from the front 
line employees in a confidential and non-punitive environment, we can not only see 
what is happening, but also why it is happening. Today, these stand-alone safety 
programs at individual airlines are reaching their maturity. That is a reflection of 
the dynamic nature of any data collection effort — it must adapt to changes in the 
environment; in this case, the changes in the aviation industry. 

As safety professionals continue to see value in these programs and work with 
them in more detail, it has become clear that even more can be learned by sharing 
safety information among the various stakeholders in the industry. The FAA and 
the airline industry, including ALPA, continue to work together on developing a for- 
malized process in which safety information can be accessed through secure net- 
works under mutually agreeable rules of engagement. ALPA has been working 
closely with the FAA, NASA, and the airlines to develop a process that will make 
this safety information available to decision-makers to help them in their efforts to 
manage risk. This process is also invaluable in the sharing of accident- and incident- 
prevention strate^es across the industry. Again, though, I would point out that as 
time goes on, the industry continues to refine our processes for maximizing the safe- 
ty benefits that the traveling public receives from collecting data while at the same 
time protecting those employees and airlines that bring the data to the table. 

NASA, especially through the Aviation Safety Reporting System (ASRS) program, 
has always been an important player in aviation safety. Its human factors research, 
in particular, has provided great value to our industry. The National Aviation Oper- 
ations Monitoring Service (NAOMS) survey was part of the early effort to provide 
more information to help all of us improve aviation safety. This first survey was a 
test of the process and methodology. We understand that the data extracted from 
this survey were summarized and those summaries were shared with the govern- 
ment and industry. As in any first test, the data didn’t correlate very well with data 
from other sources, possibly due to the mix of general aviation and airline oper- 
ations. The aviation community had plans to further analyze those discrepancies 
and determine if the data were reliable, but funding for NAOMS ran out. That is 
when ALPA stepped in to help keep the project alive as a part of our involvement 
with the Commercial Aviation Safety Team (CAST). While we have been working 
with CAST to modify the survey, we did not receive any of the collected data from 
NASA. 

What should happen to the data now? Several solutions are available. One that 
makes a lot of sense is to provide NASA with the necessary resources so that it can 
complete a peer review of the data and then analyze the data, while at the same 
time maintain the confidentiality and protective provisions that apply to voluntarily 
supplied safety information. Other solutions may also exist. 

Regardless of the solution, it is important to keep in mind that raw data, distrib- 
uted without appropriate analysis and scrutiny to ensure its validity, can lead to 
unintended consequences. Incomplete or inaccurate conclusions can be reached if the 
collection method is flawed or if people looking at the data aren’t familiar with avia- 
tion or the context of how that information was provided. No one knows and under- 
stands the data better than the stakeholders that provide the data in the first place. 
That is why it is so important that those stakeholders work closely with the ana- 
lysts of the data. This will ensure accurate and meaningful conclusions can be 
reached. 

Just as importantly, if raw data are simply distributed to the general public with- 
out the quality controls I’ve mentioned, it would undermine the confidence that pi- 
lots and the airline community have that voluntarily and confidentially supplied 
safety data will remain secure. As an airline captain, and one who represents the 
safety interests of 60,000 other airline pilots, I’m concerned that this could very well 
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erode the very programs that have driven the excellent safety record of airline trav- 
el that the public has come to rely on. 

Thank you, again for the opportunity to testify today. I will be pleased to address 
any questions that you may have. 

Biography for Terry L. McVenes 

Capt. Terry McVenes serves as the Executive Air Safety Chairman for the Air 
Line Pilots Association, International, representing ALFA pilots in airline safety 
and engineering matters arising within the industry. His responsibilities include 
oversight of more than 600 safety representatives from 42 airlines in the United 
States and Canada, as well as budgetary and management supervision of more than 
200 projects within the ALFA safety structure. 

Capt. McVenes chairs the Steering and Oversight Committee for the ALFA Inter- 
national safety structure and is a former member of the Operations Committee and 
MMEL Working Group. He represents ALFA pilots on the FAA’s Voluntary Aviation 
Safety Information Sharing Aviation Rule-making Committee and serves as its co- 
chairman. He has spoken at many international forums on a wide variety of avia- 
tion safety topics. He has also authored numerous articles on aviation safety, which 
have appeared in national and international publications. 

Prior to his current appointment, Capt. McVenes served as Executive Air Safety 
Vice Chairman, Chairman of the Central Air Safety Committee for U.S. Airways, 
and Chairman of the Aircraft Evaluation Committee. He coordinated the establish- 
ment of the Aviation Safety Action Program (ASAP) at U.S. Airways and served as 
a member of the FOQA Monitoring Team. He has participated in numerous accident 
and incident investigations and was a member of several line safety audit teams. 
Capt. McVenes also served as a member of the Airbus Integration Team and the 
Fuel Awareness and Conservation Team. 

Capt. McVenes began his airline career in 1978 with Rocky Mountain Airways in 
Denver, Colo., flying the DHC-6 (Twin Otter) and DHC-7 (Dash 7) aircraft. In 
March 1985, he was hired by Pacific Southwest Airlines (PSA), which later merged 
into US Airways. He is rated on the DHC-7, BAe-146, FK-28, DC-9, MD-80, A- 
320, and B-737. He currently is a captain on the A320 for U.S. Airways and has 
more than 17,000 hours of flying time. 

Prior to his airline career, Capt. McVenes was employed as an engineer for the 
Boeing Company in Seattle, Wash. He holds a Bachelor of Science degree in aero- 
space engineering from the University of Colorado and the certificate of aviation 
safety management from the University of Southern California. 


Discussion 

NAOMS Survey and Methodology 

Chairman Gordon. Thank you, Captain McVenes. 

Dr. Krosnick, is it fair to summarize a portion of your testimony 
by saying that when the methodology and the program was set up, 
the NAOMS Program, that it was set up in a way that the con- 
fidentiality of the material would be protected? 

And if that was the case, and I think that, again, NASA certified 
that when they said that they set it up by saying, we have no 
means for — anyway — they assured us in their report that that 
would be the case. So how long should it take them to get that in- 
formation to us? 

Dr. Krosnick. I would think less than a week to assure that any 
incidental open-ended responses in the file don’t happen to mention 
an airport or an airline. And the Director mentioned the idea of 
eliminating fields in the data set. I would think the normal federal 
procedure would be to redact words rather than entire fields of 
data. 

Chairman GORDON. Well, I would hope that NASA would hear 
your testimony and that the end of the year is a worst-case sce- 
nario and next week is a best-case scenario. 
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Also, Dr. Krosnick, the purpose of the NAOMS was to go heyond 
the event driven or so-called action response syndrome to aviation 
safety and develop a statistical, valid database for safety-related 
events for decision-makers. It was specifically designed to overcome 
the shortcomings of the voluntary anecdotal Aviation Safety Re- 
porting System, which couldn’t be used to tell anyone how often 
certain events occurred. 

Is that accurate? 

Dr. Krosnick. Yes. That is exactly right. That, the ASRS System 
relies on pilots to voluntarily choose to fill out a form and mail it 
in when they feel an event has occurred that merits that. And cer- 
tainly plenty of forms are filled out and mailed in every year, but 
because it is voluntary, there is every reason to believe that many 
events that occur do not get reported through that system. 

So the purpose of NAOMS was to assure that with a representa- 
tive sample of pilots who were interviewed every week of every 
year, that it would be possible to count up events in many cat- 
egories that never get described in reports to ASRS. 

Chairman Gordon. And was it successful in doing so? 

Dr. Krosnick. Well, we can’t quite answer that question, can 
we? What we know is that we designed — I should say the team de- 
signed with my help a superb methodology and implemented it 
with the approval of 0MB, which is a pretty tough critic of survey 
methods in the Federal Government, and so we can believe in the 
method, but when the data come back, the next step is to analyze 
those data fully, write reports, have those reports peer reviewed, 
and proceed ahead with assessments of validity, which we would 
have loved to do if the funding hadn’t been shut down early. 

Chairman Gordon. Well, it seems to me that this was an ex- 
traordinary high percentage of return. And you mentioned, what 
did you, was it 40,000 commercial pilots? 

Dr. Krosnick. Twenty-four thousand commercial pilots inter- 
viewed. 

Chairman Gordon. Right. I understand that, but how many are 
there in total? 

Dr. Krosnick. Oh, in the population? 

Chairman Gordon. Yes, sir. 

Dr. Krosnick. I will defer to Bob Dodd on that. 

Chairman Gordon. Or maybe Captain McVenes. Approximately 
what number of commercial pilots are there? 

Captain McVenes. There is probably roughly 100,000 commer- 
cial pilots. 

Dr. Krosnick. That is the number that we worked with. 

Chairman Gordon. So, you know, it is, to me a fourth that re- 
sponded voluntarily is an incredible number and should be 

Dr. Krosnick. Well, if you don’t mind, let us be careful about 
that. 

Chairman Gordon. Okay. 

Dr. Krosnick. It is actually not 24,000 pilots who were inter- 
viewed. It is 24,000 interviews were conducted. So we drew statis- 
tical samples of very small numbers of pilots to be interviewed each 
week. 

Chairman Gordon. How many would you say, how many dif- 
ferent pilots would have been interviewed? 
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Dr. Krosnick. About 8,000 a year. 

Chairman Gordon. Which is still an exceptionally large sam- 
pling. 

Dr. Krosnick. Yeah. Much bigger than most surveys. Absolutely. 

Chairman GORDON. And was it intended to be a continuing per- 
manent database or just a short-term experiment? 

Dr. Krosnick. Well, the slide that I showed earlier that NASA 
displayed at all the public meetings that we did early on indicated 
that it was planned to he a permanent monitoring system. 

Chairman Gordon. Well, then I hope that we get it up and run- 
ning. I think it — again, let me, once again state that the United 
States of America has the safest air transportation system in the 
world, and I think part of that reason as Mr. Hall said earlier, was 
because of the transparency, of continuing to try to do things bet- 
ter, better, better, better, and this is just one more effort to raise 
that extraordinarily high bar or I won’t say raise it any higher but 
keep it there. 

I thank you, and Mr. Hall is recognized. 

Survey Methodology and Confidentiality 

Mr. Hall of Texas. Thank you, Mr. Chairman. Captain 
McVenes, you said regardless of the solution it is important to keep 
in mind that raw data distributed without appropriate analysis and 
scrutiny to ensure its validity can lead to unintended consequences. 
Actually, sir, we have heard from several researchers that commer- 
cial and general aviation pilots were very receptive and even were 
very eager to share their experiences and views with NASA re- 
searchers in part because they were told that they would be anony- 
mous and would be protected. 

So how confident are you that releasing the data with confiden- 
tial information removed as described by Administrator Griffin will 
not hinder pilots from participating in future surveys? 

Captain McVenes. Well, the confidentiality piece is so very im- 
portant. 

Mr. Hall of Texas. Very important. 

Captain McVenes. Because it is what makes that transparency 
happen. It makes people want to report knowing that that informa- 
tion is going to be used pro-actively in a safety-related type of ac- 
tivity as opposed to some other activity of any sort of sensa- 
tionalism or whatever it may be. So that is why it is very impor- 
tant to keep that flow of information coming, and the reason that 
we have been successful as an industry to get a lot of voluntary 
participation in these programs, whether it is the NAOMS survey 
or the individual programs that are going on at our airlines, is be- 
cause that information is used pro-actively. It is not used in a puni- 
tive type of environment. It is used for safety purposes. And that 
is why that is so important. 

Mr. Hall of Texas. And Dr. Dodd, you and Dr. Krosnick were 
shaking your head indicating that you agree with his 

Dr. Dodd. That is correct. 

Dr. Krosnick. Yeah. I think it is very important that respondent 
confidentiality 

Mr. Hall of Texas. Yeah. It certainly makes sense. 

Dr. Krosnick. — never be compromised. 
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Mr. Hall of Texas. Sure. 

Dr. Krosnick. And the good news is for everyone here that the 
survey data were collected in a way so that no one could identify 
the pilots. In other words, the data are in electronic files that do 
not have the identities of the pilots in them. And so there are 
24,000 rows of numbers indicating the answers that they gave to 
statistical questions but not in any way indicating their name, 
phone number, or identity in any other way. 

So that is the good news. 

Mr. Hall of Texas. And I think the Chairman in his inquiry to 
you asked you in your testimony you state that NAOMS was al- 
ways envisioned to continue operating, and whether or not this was 
planned to be continued at NASA or at another government agency 
like the FAA. How was it? Who did finish it that last year? 

Dr. Krosnick. Well, in the — all of the work on NAOMS to date 
has been done by NASA, and so my understanding is that there 
was a planned attempt at a hand-off of the methodology to ALFA. 
The plan as you have heard already from the Chairman was to 
switch from telephone interviewing, which we had determined to be 
the most reliable way to make these measurements, over to Inter- 
net data collection, where respondents could go to a website and 
answer questionnaires there. 

Unfortunately, a test of that methodology was carried out by 
NASA, and as I understand it was unfortunately quite a failure, 
that hardly any pilots participated in the Internet version of the 
survey. And I am not surprised by that, because our research 
methods literature suggests that respondents of this sort are far 
more likely to participate if the telephone call method is used. 

So my personal concern at the moment is the only plans I have 
heard for ALFA possibly to pick up this project are with this meth- 
odology which has already shown to be not feasible. But more im- 
portantly I guess I share perhaps the implication of what you are 
suggesting, and that is that I don’t know that this is an operation 
that can work effectively outside of government. And I think it is 
particularly important to recognize, as I said in my comments, that 
NASA is really trusted by pilots, as I am sure Capt. McVenes will 
acknowledge, because the ASRS has been so successful in collecting 
very detailed information that is made public and that reveals a lot 
about the details of bad things that go on. We heard earlier a tran- 
script of a pilot talking about falling asleep in the cockpit. That is 
a pretty scary story, and that is on the Internet for anyone to read. 

And so, you know, the possibility that that information being re- 
vealed to the public and its benefits seems clearly to outweigh the 
possibility that someone could get in trouble because NASA has 
successfully protected people from that. And I believe NASA has 
the trust and credibility with pilots to continue to do that. 

Why Didn’t the FAA Continue the Froject? 

Mr. Hall of Texas. Doctor, thank you. I will ask any of the 
three of you, do you all know why FAA didn’t pick up the project? 
Why didn’t they pick the project up? 

Dr. Dodd. Well, I don’t think we originally planned for FAA to 
pick up the project, and what Dr. Krosnick was addressing is key 
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to that issue, and that is NASA has a reputation among the pilot 
community of protecting their identity. 

The Aviation Safety Reporting System, which you have heard 
referenced a number of times today, is a program that has been in 
existence for 30 years. During that time not one pilot’s confiden- 
tiality has been compromised. There has never been any reverse 
engineering where somebody has gone into the report and been 
able to identify who the reporter was, and because of that NASA 
was chosen to be the primary and best government agency to do 
this work because of that reputation. 

The FAA’s mission is slightly different, and of course, the FAA 
is responsible for enforcement and certification of pilots. And be- 
cause of that, pilots may be unwilling to voluntarily report issues 
that might result in them getting a slap on the hand, if you will, 
or what we call a certificate of action. 

So historically surveys run by the FAA among the pilot commu- 
nity don’t have a very high response rate, which is one of the 
metrics that we use to evaluate how well we are doing with the 
survey. 

As an aside, with this particular survey that NAOMS, that 
NASA did with NAOMS, we had an 85 percent acceptance rate 
among the pilots contacted who agreed to do the survey. That is 
an exceptionally high response rate and gives us confidence that 
the pilots were willing to meaningfully engage in the process. 

Mr. Hall of Texas. Thank you. My time really has expired. I 
yield back any time I have or don’t have. 

Chairman Gordon. Not much, Mr. Hall. 

The Space and Aeronautics Subcommittee Chairman, Mr. Udall, 
is represented or recognized. 

Best Organization to Operate NAOMS 

Mr. Udall. Thank you, Mr. Chairman. Again, I want to thank 
the panel for your compelling and insightful testimony. 

Dr. Dodd, if I could focus on part of your testimony to start with, 
you stated, I want to get this right. I believe that NAOMS should 
be reinstituted and operated by an independent and unbiased orga- 
nization. Should NAOMS be operated by NASA, some other organi- 
zation? What would you recommend? 

Dr. Dodd. I think there is a number of suitable organizations, 
and it would depend on a number of issues. I think Dr. Krosnick’s 
observation that this is inherently a government type of activity is 
absolutely correct. So I would not hazard to recommend what agen- 
cies might be appropriate. I think NASA at the working staff level 
did an outstanding job with this project, and they have the tech- 
nical expertise to do that. So I certainly would have no objections 
from NASA continuing to do this work. 

So certainly that would be one agency that fits my definition. 

Mr. Udall. Do the other panelists care to comment on that ques- 
tion? 

Dr. Krosnick. Yeah. I agree, of course, that NASA is suitable as 
I have suggested already in my comments, and from the extensive 
learning I have benefited from about the airline operation and in- 
dustry, it is hard for me to identify another organization from my 
many hours talking with pilots that the practitioners would have 
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the same confidence in. To some degree there is specialization in 
these federal agencies, and the FAA is particularly good at col- 
lecting large electronic databases from machines. 

NASA’s specialty in this area has been activities like ASRS 
where humans are reporting their experiences. So to some degree 
NAOMS fits very nicely under the NASA umbrella. And I agree 
with Dr. Dodd that NASA has done a wonderful job with this 
project and has earned the recognition that they deserve, I think, 
by that quality of work, and why not let them continue. 

Captain McVenes. And certainly from our perspective, you 
know, having NASA continue with the project is — we certainly 
wouldn’t object to. Our role in the whole thing was to keep it alive 
in whatever way we could. And to Mr. Hall’s point as, you know, 
why FAA didn’t take it over directly, kind of indirectly they were 
involved with it in the fact that their work with the commercial 
aviation safety team, as well as the rest of the industry, we were 
trying to utilize that group as a way to keep this thing going and 
involve all the stakeholders including the FAA on this. 

Termination oe Program 

Mr. Udall. If I might return to you. Captain for a second, final 
question, editorialize briefly, and my colleague. Congressman Li- 
pinski asked a question in the earlier round, the first panel, what, 
why did NASA stop? What was underway here, and it is curious, 
but sitting that aside, I want to thank you. Captain, for your will- 
ingness to testify on such short notice in front of the Committee. 

Captain McVenes. My pleasure. 

Mr. Udall. And I was struck by one of your statements which 
read, NASA has always been an important player in aviation safe- 
ty. It is human factors, research in particular, that provide a great 
value to our industry. At some committee hearings that I have 
chaired earlier this year we have heard numerous concerns raised 
about the cutbacks and the NASA human factors R&D programs 
in recent years, particularly in the applied areas. 

Have you heard these same concerns raised, and do you share 
them, and after the Captain is finished, if the other two panelists 
would care to comment, I would sure appreciate it. 

Captain McVenes. Yeah. We were concerned. I know we wrote 
several letters from our president to the various groups here in 
Washington to try to change the mind of those that controlled the 
purse strings over that, because we saw a great value in human 
factors research that was going on, especially as it applied to some 
of the automation, changes that were taking place in our aircraft, 
and so that is why we were very interested in trying to keep that 
alive as best we could. 

This is an important part of aviation, especially the future of 
aviation as we continue to evolve with new technologies, we under- 
stand what that human element is in the role of how we fly our 
airplanes. And NASA played a very big role in that in the past, and 
unfortunately, they are not doing it as much anymore as they — we 
feel they should be. 

Dr. Krosnick. I agree, and if you look at the slide that is still 
up on the screen, you will see that was an ambitious work plan for 
a great deal of research to be carried out over a long period of time. 
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and the budget that was established for that work in the beginning 
was appropriately generous to cover the cost at an A-plus quality 
level. 

But that budget shrank regularly during the years and con- 
tracted for reasons we were not informed about, such that in the 
end there was not money available to pay for most of the work to 
be done. And it was that sort of choking off of the project that ac- 
counts for the incompletion of the work. 

And I think, you know, you are perhaps pointing to a larger issue 
at NASA about changing priorities and changing budgets in a way 
that Bob is actually even more informed about than I am. 

Dr. Dodd. The only additional 

Mr. Udall. Mr. Chairman, if I might use of Mr. Hall’s remaining 
time for Dr. Dodd to comment. 

Dr. Dodd. Very quickly. The NASA human factors program, we 
saw it while we were involved with the NAOMS Project of year by 
year having funding removed from the program, and we saw it, and 
at the local level and saw that pain that it caused among the staff 
at NASA Ames. 

The other thing I want to point out is that aviation is an incred- 
ibly labor-intensive activity. I won’t go through all the activities 
that are involved with it, but human factors is key. It is usually 
human error that is associated with most of our major problems, 
and we need to continue to fund that research and that focus on 
that because it is not going to go away. 

Mr. Udall. Thank you. Thank you, Mr. Chairman. 

Chairman GORDON. Sir, your time has expired, and now the Vice 
Chairman of the Science and Technology Committee, Mr. Lipinski, 
is recognized for five minutes. 

Mr. Lipinski. Thank you. Chairman Gordon. I want to thank all 
of our witnesses for their testimony today, especially Captain 
McVenes. As Mr. Udall said, I know you did this on short notice. 
We appreciate that. 

Dr. Krosnick, I am not sure if you remember 14 years ago, I 
think it was, I did the summer program, political psychology, at 
Ohio State University. 

Dr. Krosnick. That is why you look familiar. There we go. 

Mr. Lipinski. And so I have known you for, going back many 
years there, and I certainly have a great deal of respect for your 
work. 

I wanted to sort of keep going down the line of what I started 
on earlier with the first panel. I asked them to put the timeline up 
there. I ask you. Dr. Krosnick, because I think I — actually I heard 
this earlier. I was in my office listening to the testimony, and when 
I heard this, I decided I had to come and run back here to ask 
some questions. 

Where did the process stop in this timeline? 

Dr. Krosnick. I think Dr. Dodd is the best person to describe it. 

Mr. Lipinski. Okay. 

Dr. Dodd. We basically — 2003, is when we really had the plug 
pulled on us. We — one of the things I should clarify is that NASA 
had a five-year program from a budgeting point of view for this 
project and many others, and so when you hear NASA saying that 
there was an end point for this particular project, it was because 
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of a five-year budgeting exercise and that the project was not con- 
tinued outside of that budget for the next cycle. 

It stopped in 2003, essentially, as far as continuing development. 
In 2002, we were getting ready for air traffic controllers. We did 
three focus groups with 15 air traffic controllers each, and we had 
about a year and a half development cycle planned for the develop- 
ment of the air traffic controller questionnaire. We briefed NASA. 
They were very receptive to the idea, and at that point is when we 
stopped ADC development, and it was because of the funding 
issues clearly were going to be cut back at that point. And so we 
dropped that out of the plan at that point. We didn’t have the 
money for the development. We focused on continuing the pilot sur- 
vey. 

Mr. Lipinski. That is even more information than I was aware 
of, but it fits perfectly into my question, and none of you, I believe, 
can answer this but I have questions that I asked Dr. Griffin, and 
I think he was mistaken about there not being — the program not 
being interrupted at a certain point, that it had, you know, gone 
its full course. 

Certainly there have been issues involving air traffic controllers 
and the FAA. That is a major issue, something that we have been 
dealing with, trying to get dealt with in the Transportation Com- 
mittee. It is a big labor issue, and it seems to me that what Dr. 
Dodd just said seems to fit with possibly when it was time to actu- 
ally go and do the survey of the air traffic controllers, that is where 
this stopped. 

And so I really would like to, Mr. Chairman, I think that is an 
important point to look at because what it comes down to is safety 
is the most important thing, and the whole purpose of this was for 
safety. It is $11 million, it is six years that was spent on it, but 
what can we do to improve safety. 

I am not going to say the sky is falling literally, but as you said, 
Mr. Chairman, it is just trying to make a safe system even safer, 
and I just want to leave out there the, you know, I don’t know if 
anyone — if Dr. Dodd or anyone else has any other comments on 
that, but the possibility of there got to be a place where the FAA 
perhaps did not want to go with the survey of the air traffic con- 
trollers at that time, and that is where this stopped. 

Now, if anyone wants to add anything to that or we would just 
leave it there. So any witnesses want to add anything to that? 

Dr. Krosnick. I really can’t comment because I didn’t know what 
the FAA decision-making was on that or senior NASA management 
as far as funding decisions. I am sure that there were probably 
other issues as part of that process, and other than that I can’t 
comment. 

Mr. Lipinski. Thank you. I will leave it at that, Mr. Chairman. 

Chairman GORDON. Thank you, Mr. Lipinski, and my thanks to 
our witnesses today for their time and expertise and Dr. Krosnick, 
I hope you will make yourself available and your expertise to 
NASA if they need you to help do this final, you know, cleaning if 
there is any need to be of this list. 

And if there is no objection, the record will remain open for addi- 
tional statements from the Members and for answers to any follow- 
up questions the Committee may ask of the witnesses. 
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Without objection, so ordered. 

[Whereupon, at 4:05 p.m., the Committee was adjourned.] 
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Answers to Post-Hearing Questions 

Responses by Michael D. Griffin, Administrator, National Aeronautics and Space Ad- 
ministration (NASA) 


Questions submitted by Chairman Bart Gordon 

Ql. Please provide a full discussion of the transfer of the NAOMS methodology to 
the Air Line Pilots Association (ALP A), including revisions in the questionnaire, 
ALPA’s contribution to the cost of the transfer and revisions, and whether any 
peer review was conducted on either the original survey methodology or the re- 
vised methodology as transferred to ALPA. If the NAOMS methodology was not 
peer-reviewed, please describe what process was used to validate the method- 
ology prior to transfer to ALPA. 

Al. Transfer of the NAOMS Methodology to ALPA 

The NAOMS team adapted the computer-assisted, telephone interview process 
(original survey methodology) to a weh-hased, data collection mode (revised survey 
methodology) using commercial, off-the-shelf (COTS) software (ILLUME hy DatStat 
Inc.). NASA conducted testing during the months of February and March 2006 to 
compare the weh-hased survey process with the original computer-assisted tele- 
phone survey process. 

The NAOMS team purchased a one-year license starting in December 2006 from 
DatStat to apply ILLUME technology to a web-based survey that replicated the 
functionality of the original computer-assisted telephone survey. NASA transferred 
this license to ALPA in January 2007. 

The NAOMS team provided training sessions for the ALPA team on the NAOMS 
web-based survey methodology. 

NASA has asked the National Academy of Sciences to assess the NAOMS survey 
methodology, and to the extent possible, to assess the potential utility of the survey 
responses. 

Revisions in the Questionnaire 

There were revisions to the content of the questionnaire associated with adapta- 
tion for web-based surveys. Modifications were made to the computer-assisted tele- 
phone interface model to adapt the questions so they would capture the same data 
via the web-based interface model. In addition, some questions were modified to 
simplify the telephone survey questions for the web-based survey application. 

ALPA’s Contribution to the Cost of the Transfer 

ALPA is estimated to have contributed approximately one work-year equivalent 
to support the transfer of the web-based survey methodology. 

Peer-Review of the NAOMS Methodology 

From 1998 to 2004, the NAOMS project team gave approximately 17 PowerPoint 
briefings to various audiences, mainly government and industry personnel. (These 
briefings have been provided to the House Committee on Science and Technology 
at their request.) However, none of the research conducted in the NAOMS project 
has been peer-reviewed to date. PowerPoint briefings to stakeholders, while having 
some value, do not constitute peer review. Accordingly, no product of the NAOMS 
project, including the survey methodology, the survey data, and any analysis of 
those data, should be viewed or considered at this stage as having been validated. 

It should be noted that NASA’s assertion that none of results from the NAOMS 
project can be considered validated does not mean that NASA is drawing conclu- 
sions about the validity of the survey data; we are simply stating that no such con- 
clusions can be credibly drawn. That said, comparisons of some of the results re- 
ported in the briefings prepared by the NAOMS project team to event rates that 
are known with reasonable certainty, such as engine failures, have led NASA to con- 
clude that there is reason to question the results presented by the NAOMS project 
team in their various briefings. 

In order to rectify this situation as best as possible, NASA has asked the National 
Academy of Sciences to assess the NAOMS survey methodology, and to the extent 
possible, to assess the potential utility of the survey responses. 

Q2. Please provide a breakout by fiscal year by recipient of the $11.3 million you 
stated in your testimony was spent on the NAOMS project. 

A2. Battelle was the prime contractor for the NAOMS project; $11.23 million is the 
total full cost of the project. The costs break out by fiscal year as follows: 
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NAOMSFUINDING 
HISTORY 
(in thousands) 

FY 

TOTAL 

FY98 

$1,482 

FY99 

$1,563 

FYOO 

$ 968 

FYOI 

$1,207 

FY02 

$1,800 

FY03 

$1,828 

FY04 

$1,557 

FY05 

$ 511 

FY06 

$ 314 

FY07 

~ 

TOTAL 

$11,230 


Question submitted by Representative Daniel Lipinski 

Ql. I have major concerns that it is going to be difficult to make improvements in 
the aviation industry if the agencies cannot work coUaboratively and trust each 
other’s work. Dr. Griffin, could you comment on your working relationship with 
the FAA? 

Al. A solid collaborative working relationship between NASA and the Federal Avia- 
tion Administration (FAA) is critical to the successful outcome of the Nation’s vision 
for the Next Generation Air Transportation System (NextGen). The working rela- 
tionship between NASA and the FAA has traditionally been solid and continues to 
strengthen at all levels. As part of the Joint Planning and Development Office 
(JPDO), a multi-agency organization focused on developing the NextGen, the FAA 
and NASA have formed a strong partnership with a common goal of a greatly im- 
proved future air transportation system for the Nation. Both the NASA and FAA 
Administrators are members of a Senior Policy Committee (SPC) that oversees the 
work of the JPDO. Among its key activities, the Committee works to provide policy 
guidance, resolve major policy issues, and identify and align resource needs. The 
partnership to bring about NextGen encompasses not only safety research but also 
air traffic management and environmental research. Participation of both Adminis- 
trators on the SPC demonstrates at the highest level within each agency a relation- 
ship that is committed to a future aviation system that is responsive to the mobility 
needs of the public. 

To further ensure that a strong working relationship between NASA and FAA is 
promoted at all levels. Dr. Lisa Porter, the NASA Associate Administrator for Aero- 
nautics, meets regularly with senior management of the FAA to have open and 
frank discussions on matters the two agencies are jointly working. For example, 
during FY 2007, Dr. Porter and Mr. Nicholas Sabatini, the FAA Associate Adminis- 
trator for Aviation Safety, held joint meetings to monitor the progress of tech- 
nologies that were being developed by NASA and implemented by the FAA into 
what has become the Aviation Safety Information and Analysis Sharing (ASIAS) 
system. At the beginning of FY 2008, the ASIAS system successfully transitioned 
from NASA to the FAA and the aviation industry as a means to share a wide vari- 
ety of safety data pertaining to the national air transportation system. Going for- 
ward, NASA continues to develop advanced methods and algorithms for analyzing 
multiple and varied sources of safety data in order to enable the ability to discover 
safety precursors before accidents occur. In addition, NASA will continue to work 
coUaboratively with the FAA and industry to transition these new methods into the 
evolving NextGen. 

With regard to air traffic management research, NASA Aeronautics, the FAA Air 
Traffic Organization (ATO), and the JPDO are working coUaboratively to establish 
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a process to transfer technologies from fundamental research and development 
(R&D) into implementation for the NextGen. This process, which ensures research 
is sufficient and appropriate to enable NextGen, has top-level commitment from Dr. 
Porter and Ms. Victoria Cox, Vice President for Operations Planning Services, ATO. 
A coordinating committee that includes both FAA and NASA representatives over- 
sees four research transition teams that are organized around the NextGen Concept 
of Operations framework. This framework connects the FAA’s Operational Evolution 
Partnership elements with the NASA research. Teams are collaboratively working 
to plan near-term R&D transition in areas such as surface management and long- 
term transition in areas such as dynamic airspace allocation. 

As NextGen evolves to handle the projected growth in the national air transpor- 
tation system, environmental concerns, including the expected increase in noise and 
air pollution from a variety of emissions, pose a significant hurdle that must be 
overcome. The future aircraft fleet will need to include technology advancements 
that enable the growth in the air transportation system without additional impact 
on the environment. NASA and the FAA have a long history of collaborative work 
in this area. A variety of predictive tools developed at NASA have been incorporated 
into the FAA Environmental Management System and used to inform regulatory de- 
cisions. In addition, over the last year, the FAA and NASA have worked together 
on the development of the Goals and Objectives for the Energy & Environment por- 
tion of the National Plan for Aeronautics R&D. Both agencies continue to work 
closely to ensure that fundamental technology developed at NASA can be 
transitioned to the future fleet. 

Finally, NASA and the FAA actively participate in each other’s advisory/review 
committees with representatives of each agency engaging, in an advisory role, in de- 
termining the strategic directions of the research of the other. For example. Dr. Por- 
ter serves on the FAA’s Research and Development Advisory Committee (REDAC) 
which reviews and then advises the FAA senior management on the relevance and 
progress of their research and development activities. Further strengthening this 
collaboration across multiple technical areas and management levels, representa- 
tives from each of the three NASA Research Programs in Aeronautics serve as mem- 
bers on subcommittees to the FAA REDAC. In a similar fashion, and at the request 
of NASA, the FAA has provided representatives to participate on NASA review pan- 
els to assess the technical quality, performance, and relevance of NASA research 
programs. For two of the NASA programs, the designated leads of the review panels 
were FAA representatives. In addition, NASA researchers serve on various technical 
committees, such as the Radio Technical Commission for Aeronautics (RTCA) spe- 
cial committees that provide advice to the FAA on technical matters. NASA also 
makes use of FAA subject matter experts to help evaluate proposals received from 
universities and industry via the NASA Research Announcement process. These ex- 
amples of interagency participation on advisory committees, and other joint activi- 
ties across all levels, demonstrate a working relationship based on trust and respect 
for the talent and integrity between NASA and the FAA, particularly at the senior 
leadership level. Continued commitment to such a partnership is critical to the fu- 
ture success of NextGen. 

Questions submitted by Representative Russ Carnahan 

Ql. Dr. Griffin, news reports have indicated that NASA Associate Administrator, 
Thomas S. Luedtke, said that revealing the findings could damage the public’s 
confidence in airlines and affect airline profits. Do you believe that it is more 
important to keep the American people in the dark about the basic reality of 
where we are in terms of airline safety than to paint an honest portrait for our 
constituents? 

Al. The Associated Press (AP) requested the survey results from this project 
through the Freedom of Information Act (FOIA). NASA made a determination not 
to release the survey results, using an exemption available under the FOIA. I stated 
earlier, both to the public and in Congressional testimony, that I do not agree with 
the way the FOIA exemption was explained and regret any impression that NASA 
was in any way putting commercial interests ahead of public safety. That was not, 
and never will be, the case. 

Q2. In Mr. Luedtke’s final denial letter to the AP regarding its request for the survey 
results, he wrote that “release of the requested data, which are sensitive and 
safety-related, could materially affect the public confidence in, and the commer- 
cial welfare of, the air carriers and general aviation companies whose pilots par- 
ticipated in the survey ...” This seems to indicate that the results portrayed 



109 


a fairly dire assessment of air safety travel — was Mr. Luedtke going to worst 
case scenario or is this a legitimate doomsday scenario? 

A2. Mr. Luedtke’s determination to not release the survey results was neither. The 
determination had nothing to do with survey results, as no final report or conclu- 
sions had heen made. Rather, Mr. Luedtke’s letter articulated NASA’s determina- 
tion that the raw survey responses contained information protected hy the Freedom 
of Information Act (FOIA) Exemption 4, which incorporates Trade Secrets Act pro- 
tection for confidential commercial information. This exemption requires the protec- 
tion of confidential commercial information that is voluntarily provided to the Agen- 
cy and would not customarily he released to the public. Confidential commercial in- 
formation is defined very broadly and includes company information: 1) relating to 
its business, including processes, operations and statistical data; 2) which is ob- 
tained from someone outside the government; and, 3) which is not generally re- 
leased to the public. 

In response to the FOIA request from the AP, NASA cited concerns for “public 
confidence” and for the “commercial welfare” of air carriers as the supporting basis 
for the exemption cited in denying the request for the data. This sentence, though 
taken from case law, was a mistake, as NASA Administrator Griffin has made clear. 
The intent was better explained in the following sentences in the NASA response, 
which noted that the airlines and aviation industry may have a commercial interest 
in this data. It does not reflect any conclusions drawn from the data. NASA regrets 
any impression that the Agency was in any way trying to put commercial interests 
ahead of public safety. That was not, and never will be, the case. 
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Answers to Post-Hearing Questions 

Responses by James E. Hall, Managing Partner, Hall and Associates, LLC; Former 
Chairman, National Transportation Safety Board (NTSB) 

Question submitted by Representative Daniel Lipinski 

Ql. The FAA has responded to the stories on NAOMS by pointing out how safe the 
skies have been in recent years. At the same time, congestion at airports has been 
growing, we have had several near-miss collisions at airports just this year, and 
the projections are that aviation traffic will keep growing. Are the safety systems 
in place today adequate to meet the emerging challenges in aviation? 

Al. I am pleased that the question asked for comments on the adequacy of our na- 
tion’s current safety structure to address rising challenges, particularly when many 
falsely believe that our past safety successes are sufficient to guarantee continued 
success in the future. In short, Rep. Lipinski, the answer to your question is no. 

The FAA is correct in pointing out that the skies have been safer in recent years. 
In the ten year period following the 1996 Gore Commission, the airline industry suc- 
cessfully reduced fatal accidents by 65 percent. It is certainly safer to fly today than 
it was ten years ago. However, there are two major reasons why this success, while 
laudable, should not lead us to conclude that all is well in the aviation industry. 

1. Safety Requires Constant Vigilance 

The ten-year reduction in fatal accidents was the product of substantial changes — 
most of which were recommended by the Gore Commission — on the way the FAA, 
NTSB, DOT, airlines, and others handled safety and regulation. These changes oc- 
curred largely in response to two high-profile accidents and the general trends of 
rapid expansion in the industry, technological and aircraft design development, and 
large projected increases in passenger volume. 

In other words, while prior to 1996 we had an aviation safety framework — and 
though overall aviation safety had increased in the preceding 40 years — that frame- 
work was deemed no longer adequate to meet future challenges. The current safety 
of the skies that the FAA cites is therefore due to the historical commitment in our 
nation’s safety culture to resist complacency and satisfaction with existing safety 
frameworks. This commitment to constant vigilance and improvement should con- 
tinue to be reaffirmed. 

2. Nine Fatal Accidents Per Year: The Next Generation of Risks 

Today there are dangerous trends in the aviation industry that could pose serious 
safety risks if we do in fact regress to complacency. As you note in your question, 
near-miss incidents are still a major concern and congestion and volume are soaring. 
Near-misses are illustrative of the new challenges facing aviation safety. Because 
we have reduced the number of major mishaps and fatalities we must analyze such 
close-calls and nonfatal incidents in order to see if hidden dangers lurk beneath the 
surface of seemingly positive statistics. This is why the denial of the NAOMS data 
was so particularly distressing. 

Airport congestion and volume, for their part, are but some examples of what I 
call the “Next Generation of Risks,” which also includes a dramatic shortage of air 
traffic controllers, pilots, and technology upgrades. Perhaps the most significant sta- 
tistic I can find in response to your question is that cited in the February 2007 GAO 
study {Federal Aviation Administration: Challenges Facing the Agency in Fiscal 
Year 2008 and Beyond, GAO-07-490T), which stated that: 

“although the system remains extraordinarily safe, if the current acci- 
dent rate continues while air traffic potentially triples in the next 20 
years, this country would see nine fatal commercial accidents each 
year, on average.” 

Nine fatal accidents and hundreds or thousands of deaths per year would not only 
represent an annual tragedy and dramatic reversal of historical safety trends, but 
would also severely affect the confidence of the flying public — ironically, the very 
reason NASA initially provided for withholding the NAOMS data. 

Clearly, we do not currently have the safety system necessary for the next genera- 
tion of risks. The FAA estimates it will lose about 70 percent of the air traffic con- 
troller workforce over the next 10 years. In 2009, airlines will have to fill 20,000 
pilot openings due to retirements and other factors. Passenger volume is projected 
to reach one billion by 2015 and close to 2.3 billion by 2027. Numerous other poten- 
tial dangers to aviation safety also exist, but perhaps the greatest threat is the idea 
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that because we are safe now, there is no cause to worry or even think about future 
hazards. Nothing, in fact, could be more dangerous to the aviation traveling public. 

I applaud the Committee’s past and recent attention to aviation safety and I urge 
the Members to continue to exercise their vital oversight role as a driving force be- 
hind safety improvement and reform. Chairman Gordon, thank you again for the 
opportunity to be of service to yourself, the Committee, and the Congress. Please 
do not hesitate to contact me if I may be of any further assistance. 
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Answers to Post-Hearing Questions 

Responses by Robert S. Dodd, Safety Consultant and President, Dodd & Associates, 
LLC 


Questions submitted by Chairman Mark Udall 

Ql. In his testimony at the hearing, NASA Administrator Griffin compared the 
NAOMS project with the existing Aviation Safety Reporting System (ASRS), 
stating that “One of the primary differences between ASRS and this survey was 
that ASRS is managed by aviation specialists. When a report is made, the avia- 
tion specialists can contact the submitter of the report and ask follow-up ques- 
tions. They are knowledgeable about aviation safety. This (NAOMS) survey was 
conducted by telephone polling surveyors who have no knowledge or had no 
knowledge at all as to aviation or aviation safety. They had no domain expertise 
and it is precisely that which has led to some of the problems that we are dis- 
cussing today.” 

Qla. Do you agree or disagree with Administrator Griffin’s characterization^ Why? 

Ala. I do not agree with the Administrator’s characterization. There are two im- 
plicit assumptions in his statement that are in error. One relates to interviewer ex- 
pertise and the other implies that NAOMS and ASRS are similar. 

First, the Administrator implied that having knowledgeable aviation interviewers 
is preferred because it allows the interviewer to conduct follow-up questions with 
the interview subject to capture additional information, or perhaps answer questions 
if the interview subject was confused about a particular question. While on the sur- 
face this may appear to be the preferred approach, it is in reality the wrong ap- 
proach. 

It is vitally important in survey research that the questions be applied in the 
same way for each interview subject. This is a basic and fundamental characteristic 
of any quality survey. The way questions are asked matter and can influence how 
an interview subject responds. Consequently, questionnaires must be carefully de- 
signed AND interviewers trained to conduct the interview in the same way each and 
every time. 

NAOMS interviewers were not allowed to deviate in any way from the prepared 
questionnaire. The questions were designed to be clear. For the vast majority of 
questions, pilots were not confused and did not ask for clarification. For those few 
questions where pilots did ask for clarification, the NAOMS team prepared scripted 
responses in advance for the interviewers to use for the most common clarification 
questions. That was the only acceptable response if a pilot asked a question. 

NAOMS interviewers were professional interviewers who had extensive experi- 
ence in conducting interviews. They were trained to conduct the NAOMS interview 
over three separate sessions lasting a total of 12 hours. Each interviewer was then 
certified to conduct the interviews through simulated interviews with NAOMS avia- 
tion experts posing as pilots. As mentioned, their performance was also randomly 
monitored by their managers. 

NAOMS interviewers were not aviation experts and this was preferred. The 
NAOMS team did not want the interviewers to offer impromptu responses to pilots 
if they asked questions about the survey or a particular question. The goal was for 
each question to be asked the same way each time it was applied. Aviation knowl- 
edge was not required for this to occur. What was required was professional and 
disciplined interviewers experienced in conducting telephone interviews. NAOMS 
interviews were also randomly monitored so managers could ensure this basic tenet 
was being followed. 

The second assumption that appeared in the Administrator’s statement is that 
NAOMS and ASRS are in some way compatible. The programs are similar in that 
they both collect data on aviation incidents from pilots but they are very different 
in their design and goals. 

The ASRS is a voluntary reporting system where the PILOT INITIATES the con- 
tact with NASA to report an incident they experienced. NAOMS is voluntary report- 
ing system where the PILOT IS ASKED to voluntarily provide information on inci- 
dents he or she may have experienced. ASRS is designed to collect information on 
a SPECIFIC INCIDENT while NAOMS is designed to collect information on the fre- 
quency of occurrence of a BROAD RANGE OF INCIDENTS. 

ASRS data cannot be used to estimate the frequency of safety events in the Na- 
tional Airspace System (NAS) but ASRS reports are very useful in understanding 
why a particular event occurred. NAOMS on the other hand was designed to provide 
accurate estimates on how often events occurred and to measure changes in event 
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rates over time but NAOMS was not designed to collect data on why events oc- 
curred. NAOMS was designed to provide a method for the ongoing systematic collec- 
tion, analysis and interpretation of the frequency of incidents in the NAS for use 
in assisting the planning, implementation and evaluation of the Nation’s air safety 
system. 

Qlb. If the NAOMS project ere to be restarted would there be any changes that you 
think should be made either to the methodology or implementation of the 
projeet? 

Alb. The biggest issue that would need to be addressed would be the establishment 
of an advisory board and working group. The advisory board would address strategic 
issues such as funding, operating agreements among organizations and oversight of 
the program. The working group would provide guidance on survey methodology and 
application, data analysis, publications and recommendations to the aviation indus- 
try. Both of these organizations would need to support the program and believe in 
its value. NAOMS NASA staff tried to engage the aviation community and encour- 
aged the development of a working group. This was not successful. That lack of suc- 
cess may have been related to fear of the results, lack of confidence in survey meth- 
ods, and lack of certainly on how the data might be used. In any event, this in my 
opinion was the key factor that doomed NAOMS to failure. 

The second largest issue that would have to be addressed would be establishment 
of a reliable funding stream for NAOMS not subject to the variations in agency 
budget cycles. (It is assumed that NAOMS would be operated by a government enti- 
ty.) Ideally, NAOMS should receive funding directly from Congress until it was fully 
accepted and integrated into the Nation’s aviation safety oversight system. Reduced 
funding once the system was operating would cause a compromise in data quality 
and usefulness. This would likely happen if NAOMS was part of an Agency’s budg- 
et. 

The last issue that would need to be addressed would be revisions to the NAOMS 
survey process. The questionnaire for the Air Carrier pilots is mature and well vet- 
ted but it should be reviewed for acceptance by the working group and modified ac- 
cordingly (without compromising technical accuracy). The General Aviation pilot 
survey would require more work to ensure it was measuring safety incidents as in- 
tended. Finally, development work would have to be initiated to include other avia- 
tion safety stakeholders like maintenance technicians, air traffic controllers and oth- 
ers. 

Questions submitted by Representative Daniel Lipinski 

Ql. Peggy Gilligan, the Deputy Associate Administrator for Aviation Safety at the 
FAA recently cast doubt on the survey by questioning NASA’s methodology. For 
example, she is quoted as stating that the answers in the study were not suffi- 
ciently detailed. Further, Dr. Griffin’s testimony highlights inconsistencies in the 
study as compared to surveys conducted in other ways and also calls into ques- 
tion the validity of the methodology. Dr. Dodd, your testimony explains that the 
process was meticulously designed and very thorough. Could you elaborate on 
your work on the survey and explain why others may call the study into ques- 
tion? 

Al. It is difficult for me to respond to Question One since the criticism of the 
NAOMS questions and study methodology are offered in the abstract without spe- 
cific citations or examples. It should be noted that both FAA and NASA manage- 
ment had numerous opportunities to review and comment on the NAOMS questions, 
the program design and the associated methodology. These opportunities were af- 
forded the FAA and NASA through two industry workshops, numerous industry 
briefings, and program reviews. Critical comments and questions were offered by 
NASA and FAA and the NAOMS team was responsive. 

I think FAA and NASA criticisms however highlight two failings of the NAOMS 
team and the aviation industry at large. Ideally, detailed criticisms should have 
been vetted and discussed within the context of a vibrant and engaged industry 
working group so that such concerns could have heen addressed while the program 
was operating. This type of procedure would have resulted in a stronger product. 
This didn’t happen because no ongoing working group was ever successfully estab- 
lished and functional. The failure was NASA’s inability to establish an engaged 
working group that was supportive of the project. NASA staff tried but the aviation 
industry was not supportive. 

The second failure was not having the NAOMS questionnaire and underlying 
methodology reviewed and critiqued by survey methodology experts not affiliated 
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with the NAOMS team or the aviation community. This demonstrated a certain 
naivete on part of the NAOMS team. Such a review should have been accomplished 
by experts who could comment knowledgeably on survey program development and 
design, questionnaire development, data security and respondent anonymity and 
other issues. This wasn’t done and consequently, the NAOMS team continues to re- 
spond to criticisms of the survey design and methodology by organizations not well 
versed in such issues. 

While NASA and FAA are certainly entitled to their opinion, their organizational 
expertise does not lie in survey research. Criticisms of the NAOMS methodology 
should be considered within the context of the background and knowledge of those 
offering the criticism. 

Q2. How would NAOMS data be used by an aviation safety expert to improve safety 
in the skies? 

A2. NAOMS was modeled after a public health epidemiologic surveillance system. 
The Centers for Disease Control and Prevention (CDC) states a surveillance system 
is “the ongoing systematic collection, analysis and interpretation of . . . data for use 
in the planning, implementation and evaluation of public health practice.” ^ In the 
case of NAOMS, “public health practice” could be replaced with the term “aviation 
system safety.” 

NAOMS was designed to accomplish two different tasks. First, it was designed to 
reliably track aviation safety incident trends. This was accomplished by asking a 
routine set of questions that remained constant over time. If a “statistically valid” 
increase in a particular response (trend) was noted then the appropriate safety ex- 
perts in government and industry would determine if the trend was of concern. If 
so, then an appropriate supplemental investigation would be initiated to determine 
why the trend was changing. 

Tracking trends would allow safety experts to recognize changes in the aviation 
system before losses occurred. Additionally, NAOMS event trending would allow 
aviation safety experts the ability to measure the positive effects of safety enhance- 
ments. If a particular safety enhancement was working, reported events (trends) as- 
sociated with that issue should decrease. 

In addition to the ability to accurately measure and track safety incident trends, 
NAOMS was also designed to collect information on targeted or special topics. These 
would have been small focused data collection efforts on particular topics of interest. 
The NAOMS questionnaire was designed to be flexible so questions could be added 
to evaluate a particular topic such as the introduction of a new technology or new 
procedure. Data would be collected for a specific period of time (determined by the 
need) and evaluated. Once the data collection and associated evaluation was com- 
pleted, the data collection for that topic would stop and questions for a new topic 
added if needed. The ability to trend data over time, and to evaluate specific issues 
relatively quickly,^ is a very powerful combination for safety oversight. 

The NAOMS team envisioned NAOMS trend analysis to be an automated and on- 
going process. Evaluation of the trends would be done regularly with exceedance 
limits set so notification of meaningful changes would be automatic. Manual review 
of the results would occur monthly. The industry working group and other inter- 
ested parties would receive regular updates and immediate notice if worrisome 
trends emerged. Regular meetings of the working group were envisioned for review 
of the data. Publication of annual reports summarizing the data collected over the 
previous year was also planned. 

NAOMS was designed to be an early warning system and a method by which to 
collect targeted safety information quickly, reliably and cheaply. NAOMS was never 
designed to replace current safety initiatives but to supplement current information 
systems and provide capabilities currently not available. 


1 Thacker SB, Berkelman RL, Public Health Surveillance in the United States, Epidemiolog- 
ical Review, 1988; 10:164-90. 

2 The NAOMS team estimated that a special topic section could be added to the questionnaire 
in about three months. 
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Answers to Post-Hearing Questions 

Responses by Jon A. Krosnick, Frederic O. Glover Professor in Humanities and So- 
cial Sciences, Stanford University 

Questions submitted by Chairman Mark Udall 

Ql. In his testimony at the hearing, NASA Administrator Griffin noted that a “2004 
Review of NASA’s Aerospace Technology Enterprise by the National Academies 
concluded that there was not a compelling argument for continued independent 
data collection in the NAOMS project.” He went on to quote the Review as stat- 
ing that the “NAOMS Project seems to be developing a methodology to establish 
trends in aviation safety performance that are already available through other 
sources within industry and government.” Do you agree with the National Acad- 
emies assessment^ If not, why noH 

Al. The National Academies Panel spent about one hour with the NAOMS team 
amidst a long visit they paid to Mountain View, California, to collect information 
on an array of projects in addition to NAOMS. The Panel did not receive a detailed 
briefing on the NAOMS methodology, its development, or the data that had been 
collected. Thus, the panel was limited in its ability to fully understand the project. 

In its written report, the Panel stated the following: 

“NAOMS consists of a longitudinal survey of aircraft operators, gather informa- 
tion about safety-related experiences of pilots, cabin crews, and maintenance op- 
erators for both general aviation and air carriers. ... It provides statistically 
reliable results about the frequency of occurrence of safety-related incidents.” 
(An Assessment of NASA’s Aeronautics Technology Programs, 2004, p. 100). 

“The NAOMS approach is built on research and implementation of national sur- 
veys such as those of the Bureau of Labor Statistics. The NAOMS sampling 
methods have been grounded in sound interview polling science.” 

Thus, the Panel believed that the NAOMS project was intended to include data 
collection not only from pilots but also from flight attendants and mechanics. And 
the panel recognized that the NAOMS methodology was well established and cred- 
ible. 

The Panel did not conclude that NAOMS should be terminated. Instead, they rec- 
ommended that “NASA should combine the National Aviation Operations Moni- 
toring Service methodology and resources with the Aviation Safety Reporting Sys- 
tem program data to identify aviation safety trends.” 

The Panel did express concern about the issue of potential redundancy with other 
data sources but mentioned only one instance of such overlap: engine shutdowns, 
which are tracked by the FAA. The Panel did not provide a thorough analysis of 
the extent of such redundancy. 

In fact, there was a very small degree of such overlap, and it was intentionally 
designed into the NAOMS data collection system. The purpose of this overlap was 
to allow for cross-validation of the NAOMS measurements. That is, we expected to 
find similar rates and trends in the NAOMS data as would be seen in the FAA data 
on engine shutdowns, as long as the NAOMS survey question wording exactly 
matched the specifications of the records being kept by the FAA. If we were to see 
such correspondence across data sources, that would be reassuring about the valid- 
ity of the NAOMS data. Building questionnaires with such a plan for validation is 
a normal part of designing a new questionnaire-based measurement system. 

If NAOMS were to reveal levels of and trends in event rates that corresponded 
closely with rates and trends of the same events as measured in other ways, ques- 
tions addressing these events could then have been removed from the NAOMS ques- 
tionnaires. But if NAOMS rates and trends turned out to be very different from 
those produced by different data sources, this would merit further investigation. The 
discrepancy could be attributable to inadequacy in either or both measurement 
methods, and it would be worthwhile to investigate both possibilities. 

For example, NAOMS event rates may be considerably higher than those 3 delded 
by voluntary or mandatory airline or government reporting systems because people 
must take the initiative to report events via the latter systems, and if some people 
accidentally or intentionally fail to report some events, the registered rates in the 
administrative records will be misleadingly low. Much as we might hope that em- 
ployees will fully and properly participate in all voluntary and mandatory reporting 
systems, it is possible that they do not. This possibility should not be disregarded 
when comparing NAOMS event rates to rates of the same events monitored in other 
ways. 
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In sum, the NAS Panel did note redundancy between NAOMS and other record- 
keeping systems, but only a very small proportion of events measured by the 
NAOMS questionnaires were being tracked with other methods. Indeed the purpose 
of NAOMS was to track reliable trends in types of events not being measured in 
any other ways. 

Q2. In his testimony at the hearing, NASA Administrator Griffin compared the 
NAOMS Project with the existing Aviation Safety Reporting System (ASRS), 
stating that “One of the primary differences between ASRS and this survey was 
that ASRS is managed by aviation specialists. When reports are made, the avia- 
tion specialists can contact the submitter of the report and ask follow-up ques- 
tions. They are knowledgeable about aviation safety. This [NAOMS] survey was 
conducted by telephone polling surveyors, who have no knowledge or had no 
knowledge at all as to aviation or aviation safety. They had no domain expertise, 
and it is precisely that which has led to some of the problems that we are here 
discussing today.” 

Q2a. Do you agree or disagree with Administrator Griffin’s characterization^ Why? 

A2a. Dr. Griffin was correct when he said that the ASRS is “managed” by aviation 
specialists. This was true for NAOMS as well. 

Dr. Griffin was not quite correct in saying that “the aviation specialists can con- 
tact the submitter of the report and ask follow-up questions.” The managers of the 
ASRS program do not contact event reporters. 

Instead, retired pilots and other air travel professionals are employed by ASRS 
as interviewers. These individuals routinely telephone pilots who submit reports to 
ASRS to debrief them and acquire details about the event not provided by the writ- 
ten report. This is a key feature of the ASRS data gathering system: its focus is 
not on quantitative trends but rather is on gathering rich qualitative information 
about the events that pilots choose to report. 

In contrast, NAOMS is not designed to collect such rich contextual information. 
Rather, NAOMS is designed simply to count events and track trends. It is therefore 
not necessary for telephone interviewers to have expertise in aviation, because their 
task is simply to read aloud well designed and technically correct questions to pilots 
and record the counts of events that the pilots report. NAOMS’ question wordings 
were crafted through an extensive process of pretesting to assure that they would 
be clear and understandable as administered in this fashion and would not require 
aviation expertise from the interviewers. 

In fact, it would be undesirable for the interviewers to engage in any conversation 
with the survey respondents about the events they report — doing so would violate 
one of the central premises of high quality, objective survey data collection: inter- 
viewers must read the exact same question in exactly the same way to all respond- 
ents and provide no feedback on the answers provided, so as to minimize any poten- 
tial for interviewer-induced bias. 

Nonetheless, the NAOMS interviewers did receive some training in aviation mat- 
ters from an experienced pilot before they began conducting the NAOMS interviews. 
The purpose of this training was to clarify the meanings of the questions and termi- 
nology in the questionnaire, so that the interviewers could competently handle any 
unexpected interchanges with respondents on technical issues. 

Furthermore, there is no factual basis for Dr. Griffin’s claim that lack of domain 
expertise among the interviewers “has led to some of the problems that we are here 
discussing today.” Because the job of the interviewers was to read the questions and 
record the answers accurately, lack of domain expertise could not have accounted 
for any of Dr. Griffin’s concerns about the data. 

Q2b. If the NAOMS Project were to be restarted, would there be any changes that 
you think should be made to either the methodology or implementation of the 
Project? 

A2b. If the NAOMS data collection were to be restarted, I would recommend the 
following: 

1) Conduct thorough analysis of the data collected already by NAOMS, in com- 
parison with other databases tracking some of the same events, to assess 
the quality of the NAOMS data. 

2) Restart telephone interviewing of air carrier pilots using the same inter- 
viewing methodology as was being used when data collection was sus- 
pended. 

3) Draw samples of air carrier pilots to be interviewed from the full population 
of licensed pilots. The FAA maintains an updated list of this population, so 
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the samples should be drawn from this list. A subset of this list has been 
made available to the public, but because that public subset is only partial, 
the NAOMS sample should be drawn from the full FAA list. 

4) An external advisory committee should be formed to oversee and advise on 
all data collection activities, following the example set by most major sur- 
vey data collection projects funded by the Federal Government. This com- 
mittee should be composed of a mixture of aviation and survey research ex- 
perts. Ultimately, all design decisions regarding implementation of NAOMS 
data collection should be made by the project’s Principal Investigator(s), 
based upon the advice of the advisory committee. 

5) The data that are collected each month should be released in electronic files 
accompanied by full written documentation of the data collection procedures 
as soon as possible after each month’s interviewing is completed. 

6) All releases of data should be accompanied by written documentation tell- 
ing analysts how to properly compute event rates and over-time trends. Be- 
cause the design of the survey is complex, such documentation will be use- 
ful to help assure that the public does not draw unfounded inferences from 
the data. 

7) The Principal Investigator of NAOMS should issue monthly reports docu- 
menting rates and trends in the recently collected data, modeled after the 
press releases put out by the Conference Board and the University of Michi- 
gan’s Survey Research Center documenting their monthly surveys meas- 
uring consumer confidence. 

8) Data collection from general aviation pilots should be restarted using the 
procedures that NAOMS employed prior to data collection suspension. 

9) Data collection from air traffic controllers, flight attendants, and mechanics 
should be initiated after preparatory design work is initiated and com- 
pleted. This preparatory work should include focus groups and other data 
collections to build a list of events to ask about, experimental studies to 
document optimal recall period lengths for these professionals, and studies 
to document the predominant organization of events in these professionals’ 
memories. Data should be collected from these individuals via telephone 
interviewing. 

10) In keeping with the National Academy of Sciences recommendation, it 
would be desirable to coordinate NAOMS data analysis with ASRS data 
analysis. Whenever possible, trends in ASRS reports for an event should be 
compared with NAOMS trends of the same event to explore comparability. 
Likewise, NAOMS rates should be compared with rated generated using 
any other data sources tracking a small number of events measured by both 
NAOMS and other record-keeping systems. 


Questions submitted by Representative Daniel Lipinski 

Ql. Peggy Gilligan, the Deputy Associate Administrator for Aviation Safety at the 
FAA, recently cast doubt on the survey by questioning NASA’s methodology. For 
example, she is quoted as stating that the answers in the study were not suffi- 
ciently detailed. Further, Dr. Griffin’s testimony highlights inconsistencies in the 
study as compared to surveys conducted in other ways and also calls into ques- 
tion the validity of the methodology. Dr. Krosnick, your testimony explains that 
the process was meticulously designed and very thorough. Could you elaborate 
on your work on the survey and explain why others might call the study into 
question? 

Al. I was invited to help with the development of NAOMS because the project 
sought to design surveys of the highest quality to produce the most accurate meas- 
urement possible according to best practices of survey research used throughout the 
Federal Government. 

I served as an advisor to the team that carried out the work. Specifically, I at- 
tended numerous project planning meetings and public dissemination meetings (at 
which I made presentations on the science behind the survey component of the 
project and the findings of our pretest studies). I designed a series of pretesting 
studies to ascertain (1) the optimal length of time to include in the period that re- 
spondents would be asked to describe, (2) the order in which the questions should 
be asked, and (3) whether the data should be collected by telephone interviewing, 
face-to-face interviewing, or paper and pencil questionnaires. I oversaw the analysis 
of data collected in those studies and oversaw the process of writing reports describ- 
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ing their findings. I also participated in the design and implementation of focus 
groups held with air carrier pilots, air traffic controllers, and general aviation pilots 
to build lists of safety-related events that they witnessed while working. I oversaw 
the process of conducting cognitive think-aloud pretesting interviews with air carrier 
pilots to assure that the questionnaires were understandable. And I provided advise 
on most other aspects of the study design. 

My goal in providing this advice was to be sure that NAOMS design decisions 
would yield the most accurate possible measurements. 

Many observers have raised concerns about the reliability of the NAOMS data. 
These include administrators at the FAA and administrators at NASA. 

Some expressions of concern have addressed the procedures used to collect the 
NAOMS data. These concerns were articulated prior to the public release of a full 
report by Battelle describing the procedures used to collect the data and the ration- 
ales for those procedures (as far as I know, that report has not yet been publicly 
released). It therefore strikes me as premature for anyone to offer opinions about 
inadequacies in the NAOMS procedures. 

For example. Dr. Griffin expressed concern that the NAOMS interviewers were 
not aviation experts and were not tasked with collecting detailed information about 
safety-related events through conversational interviewing. As I explained above, this 
approach to interviewing is appropriate for ASRS but not for NAOMS. Standard 
practice in high quality survey interviewing involves reading the same questions 
identically to all respondents and not offering any additional comments or impro- 
vising conversation with the respondents, so as to minimize the potential for such 
improvised conversation to bias respondents’ answers. Thus, concerns about lack of 
aviation experience among the interviewers are misplaced. 

Other expressions of concern have focused on the rates of events documented 
using the NAOMS data. For example, during his testimony, Dr. Griffin mentioned 
that NAOMS indicated that diversions to alternate airports occurred at implausibly 
high rates. Some other NAOMS critics have similarly articulated concerns that 
NAOMS rates vastly exceeded rates of the same events documented by other moni- 
toring mechanisms. 

I believe that there are at least two possible reasons for these expressions of con- 
cern. First, the NAOMS surveys were designed to 3deld multiple measurements of 
the same event, and any rate calculations must be made adjusting for this multiple 
reristering of single events. In Appendix A of this letter, I explain how statistical 
calculations must be implemented to correct for this inherent aspect of NAOMS data 
collection. 

I am concerned that this sort of calculation correction was not implemented prop- 
erly by people who have analyzed the NAOMS data to date. If so, this would lead 
to the misleading impression of event rates much higher than really occurred and 
much higher than other data sources might indicate. 

A second possible reason for concern about NAOMS rates is inadequate attention 
to the details of the wording of the NAOMS questions and the measurement being 
made by other data sources. Consider, for example, Dr. Griffin’s testimony that 
NAOMS data indicated that four times per day, a transport aircraft was landed at 
an unscheduled airport in order to deal with an unruly passenger. Dr. Griffin said 
that to his knowledge, that has happened a total of two or three times since Sep- 
tember 11, 2001. 

If such a discrepancy were really present between the NAOMS data and adminis- 
trative records of such events, it would be a basis for concern about the accuracy 
of one or both of those streams of data. But in fact, the discrepancy Dr. Griffin 
pointed to is an illusion. 

In fact, the NAOMS survey did not ask the pilots to report how many times they 
had to land an airplane at an unscheduled airport in order to deal with an unruly 
passenger. Instead, the NAOMS question asked: “During the last 60 days, how 
many times did an in-flight aircraft on which you were a crew member expedite 
landing or divert to an alternate airport due to a passenger disturbance?” Notice 
that this question combines diversions with expedited landings. It is therefore not 
appropriate to compare the total number of NAOMS reports of events in this cat- 
egory with another measuring system’s assessment of the number of times that un- 
ruly passengers caused diversions to alternate airports. Of course, the NAOMS 
question will yield higher rates than the other monitoring system will. 

These are two of the possible reasons for unfounded concerns about the accuracy 
of NAOMS data: incorrect computation of statistics using the data, and insufficient 
attention to the details of the survey question wordings and the types of events 
tracked by other monitoring systems. Mistakes of the sort outlined above would 
cause the illusory appearance of implausibly high event rates in the NAOMS sur- 
vey. 



119 


Assuming that such calculation and interpretation mistakes have been made and 
have led to a misperception that NAOMS event rates are unusually high, it is un- 
derstandable that people observing those rates might take them to be dangerous if 
released publicly, for at least two reasons. First, as the NASA FOIA appeal reply 
outlined, reports of event rates much higher than have been recognized might cause 
public concern about the safety of flying and impact the economic viability of com- 
mercial airlines. Based on my knowledge of the risk perception and decision-making 
literatures generated by social and behavioral scientists, I believe that releasing 
such numbers in the context of a public report about NAOMS is very unlikely to 
increase public fear of flying or decrease airline passenger loads. But it is certainly 
possible. So an observer within NASA or the FAA might fear negative consequences 
of releasing high rates based on NAOMS data. 

Second, staff within the FAA may perceive that event rates higher than those 
yielded by their own monitoring systems could call those monitoring systems into 
question. And in fact, that is just what higher rates from NAOMS should do, in my 
opinion. Staff members who wish to protect the appearance of integrity of those sys- 
tems might prefer that such concern not be raised in the public mind. But in my 
opinion, every measurement system is potentially subject to error, so it is always 
preferable to track important events using multiple measuring tools and to check 
their correspondence. Rather than assuming that one measuring tool is necessarily 
correct and the other is inaccurate, a discrepancy should inspire scrutiny of the im- 
plementation of both methods. Such scrutiny may lead to the detection of flaws in 
either or both measuring systems, which can in the end inspire repairs that enhance 
accuracy of assessments in the future. 

In sum, I believe that some observers may be motivated to criticize NAOMS be- 
cause of perception that NAOMS yielded implausibly high event rates. After careful 
and proper statistical calculations are implemented, accompanied by careful atten- 
tion to the exact wordings of the NAOMS questions, these rates may turn out to 
be considerably lower and may match rates of events tracked using other monitoring 
systems. 

Q2. Dr. Krosnick, you are a renowned expert on survey methodology and statistical 
analysis brought in as a subcontractor on the NAOMS project. Did the process 
used to develop the NAOMS survey instrument seem inadequate in any way? 
Did you lack expert feedback — peer review — as the methodology of the project 
went forward? 

A2. I believe that the NAOMS development process was indeed consistent with best 
practices in survey methodology. Indeed, in some ways, the preparatory design work 
exceeded that done for many major, long-standing federally funded and adminis- 
tered survey research projects. And the very high response rates that typified 
NAOMS are evidence of little if any non-response bias in the resulting data. In sum, 
I believe that NASA did an excellent job of funding top-level methodological work 
and that Battelle and its subcontractors did their work to the highest standards of 
excellence. 

The NAOMS project implemented survey methods that have been extensively peer 
reviewed and have been widely accepted as standard practice in the industry for 
decades. The tailoring of implementation of those procedures to the NAOMS context 
was also done using pretesting procedures that have well-established status in the 
methods literature. 

As I mentioned during my oral testimony, the project sought peer commentary 
and suggestions by social scientists and aviation experts at many public and private 
briefing meetings. These conversations yielded useful suggestions that influenced 
the design of NAOMS. In addition, the White House Office of Management and 
Budget reviewed the NAOMS procedure in order to approve its data collection. 0MB 
routinely evaluates federal survey project methodology and makes suggestions for 
improvement, and we benefited from this process as well. 

The only potentially valuable form of peer review that was not implemented but 
might have been helpful would have entailed forming a committee of peer reviewers 
who were paid to critique the methodology as harshly as possible and to suggest 
alternative methods to implement the survey. I believe that such a procedure would 
most likely have yielded few if any suggestions of changes to the methodology that 
was employed. But it could have been done prior to NAOMS’ suspension and could 
still be implemented today. 
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Appendix A: 


Explanation of Inflated Probabilities 
of Event Occurrences in NAOMS Data 

This Appendix explains why each event that occurs during the course of air travel 
has an inflated probability of occurrence in the NAOMS survey, by design, because 
it is witnessed by multiple people. And this Appendix explains how correction for 
this aspect of the survey design must be implemented in order to properly generate 
estimates of rates of events. 

Consider, for example, the NAOMS question asking pilots to report the number 
of times a bird hit a plane on which he/she was working. Each such bird strike 
would be witnessed by at least two pilots (the pilot and co-pilot) and could have 
been witnessed by three pilots (on aircraft with a third working cockpit crew mem- 
ber). Thus, the probability that each bird strike would be reported by some respond- 
ent in the survey was twice or three times as high as would have occurred if only 
one person had witnessed each event. And for events that involve two aircraft at 
the same time (e.g., a near miss), between four and six cockpit crew members will 
witness the event. 

Some observers have asserted that such inflated probabilities can be ignored, be- 
cause the relatively small number of pilots interviewed each month relative to the 
total population of pilots means that the chances that the same event will be re- 
ported in the survey by two different respondents is extremely small. That is true, 
but it is irrelevant to the multiple-counting issue: each event nonetheless has twice 
or three times the probability of being reported by someone. 

To illustrate how the calculation of event rates must be done, imagine that there 
are 10,000 total active air carrier pilots and that 1,000 of them were interviewed 
and asked to describe events that occurred during some or all of January, 2001. 
Imagine further that during these interviews, the total number of January bird 
strikes reported by all respondents was 50. 

To calculate the total number of bird strikes that occurred in January, it might 
be tempting to divide the number 50 by the sampling fraction (1,000/10,000), which 
would equal 500. But this would be incorrect. 

To calculate the total number of events properly, it would be necessary to use in- 
formation from the NAOMS questionnaires about the type of aircraft flown by each 
pilot who reported a bird strike to infer whether that bird strike was most likely 
witnessed by only two pilots or three pilots (this can be determined by type of air- 
craft). Then each bird strike report must be divided by the total number of pilots 
who would most likely have witnessed it (two or three) — so some bird strikes would 
contribute one-half to the total and others would contribute one-third to the total. 
Then the resulting fractions could be added up across respondents, divided by 1,000 
and multiplied by 10,000 to yield an estimate of the total number of bird strikes 
that occurred during January. 

Another calculation method would involve dividing the total number of bird 
strikes reported to have happened during January, 2000, by the total number of 
hours that the interviewed sample of pilots said they flew during that month or by 
the total number of flight legs that the interviewed sample of pilots said they flew 
during that month. These rates could then be multiplied by the total number of 
flight hours flown by all pilots during the month or by the total number of legs 
flown by all pilots during that month, respectively. But again, these numbers would 
be inappropriately high, because they would be inflated due to the doubled or tripled 
probability of reporting the same event by multiple witnesses. So again, each re- 
spondent’s report of an event should be counted as either one-half or one-third (de- 
pending on whether two or three cockpit crew were working on the aircraft); these 
fractions should then be summed, and the total should be multiplied by the total 
number of hours or legs flown by the entire population of pilots during the month 
of interest. 


1 Remember that some NAOMS respondents were interviewed on almost every day of each 
year, and they were asked to report the total number of events of each type that they witnessed 
during the past 60 days. Therefore, some respondents will have made reports for periods includ- 
ing all of January, 2001. And other respondents will have made reports for periods including 
only part of that month. Therefore, the data from different respondents must be integrated care- 
fully, to recognize the fact that some people’s reports included a mixture of days in January and 
days in other months. 
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Answers to Post-Hearing Questions 

Responses by Captain Terry L. McVenes, Executive Air Safety Chairman, Air Line 
Pilots Association, International 

Question submitted by Representative Daniel Lipinski 

Ql. When Battelle ran this project, the names of survey participants were removed 
from their records within 24 hours of the conclusion of their survey. Is that the 
kind of step that you would endorse in any other survey of this kind? 

Al. Participation in the survey by pilots was done under the assumption that it 
would be completely confidential. That premise was key to getting open and honest 
reporting. As I understand it, follow-up questioning wasn’t part of the methodology 
used in the survey and the names of the participants wasn’t germane to the type 
of questions asked. Consequently, it is my belief that future surveys of this type 
should also de-identify the participants as was done in the NAOMS survey. This 
would further promote the confidence in that confidentiality as well as provide the 
industry with quality safety information. 




Appendix 2: 


Additional Material for the Record 
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Kxhibit ^1 

factiva. 


Co6 Angelce ©imcs 


PART-A; M«tro D«tli 

Danger on ttw Ground. Too Saioty: Noar^nitaaa hava oecurod on runwaya and taxiwaya, lOdaral 
rocoids show. Plota won aomoMmot loot or coolroltara movod planaa kilo anothar'a path. 

WILUAM C. REMPEU DAVID FREED 

TIMES STAFF WRITERS 

1.2Mwo(da 

3 Fabniaiy 19S1 

Loa Angaan Tnnoa 

Soudiland 

3 

Engtnn 

(Copyngnt Tlia Tanaa Maror Company. Loa Angalaa Tanaa 1991 Al Rigldt Raoarvod) 

WMIa ma crowdad aloaa abovo Loa Angolaa imamalional Aapon hava baen raeopnizad as among lha moot 
hazaidojs ai the naeon. II la on tha grouniHn lha akporTS maza of nmwaya and tajowaya-whafa many 
acodanls and harrowing naar-misaat harva oecurrod m racaid yooia. raoorda show. 

In Jura. 1989. a Jodinar taiood onso Rurway 25-nght for takaoS ol da sarra hna anoOar ioOliiar was 
praparing to land on da Sana rvinw^. Tha landmg plana, wfaeh apparandy was oommg n on da wrong 
njnway. abortad da approach to avoid a coMon. 

On odar oocaaona, piols have baoona lost on taxrways at da Loa Angalaa aaport sdayad onto acava 
runwaya and hava loridad or oompialad MaolCs whda narrowly misaing oVar akerafl on da runways, 
a ce orang a piol and oontroHar rapona fled with dia National A a r onaudca and Spaoa Admnsiraiian and 
ravawsd Sobxday by Tha Tnas- 

Offieals of lha NaSonal Transportation Sofaty Board hava warned rapaaladly that ground acodants at busy 
U.S. odportt. Indudag in Loa Angalaa. poaa t *high pMandal lor ootasoopha.' and they hava plaoad a high 
priordy on improving ground salaly. 

Attar dia fatal oodlalon Friday balwaan an arriving USAk jaOnar and a dapanmg SkyWoat commutar 
plana, local pdols and others co mendad that given die number of laaa-aanaua groiavd aoodents and 
naar-nsaaea that prooadad d, this catasliopha was only a mallar of bme. 

No one had dad ki a oommarcial aviabon accident at lha oapoil amoe 1978. whan a HonoUu^wund 
Conbrontal Aadnas DC-tO blow two bras and made an anargoney londng altar aborting takaod. Three 
paopla wars kilad. 


A handM of other instances hava oocunad more reoandy in which planas siamcnad or sM siU aach odiar 
whila taiding. but no one has bean aoriously Inpirsd. 

Tha workTs worst aviabon ooctdani ocourrsd on a runway ki lha Canary Wands ki 1 977, vdianhwo Boeing 
Jumbo (Sts colbdad in fog, killing 583. 

From the as. Los Angalaa I m amabonal a a ralaavaly simpla alrpoil-lwo paralW sots of naiways soparalad 
by lha built of lha avports paasangar tanninalt. 

Bu! trymg to safaly plaoa asplanas in ssguanca as they land and taka off on dwsa aame nsiweya can be a 
nightmars. particularly al pa* horn such as Friday avankig. ok traffic controllars soy. 

All mamar of okoralL big and smal. fast and slow, fly m and out of tha Los Angalat akpoiL Coordktaimg 
lham rei|usas that contrrillais gauge now long it wd taka a daparbng plana to leave a runway lo avoid 
onoOiar plana that may be coming in aaoonds bahkid d. 

Tha pelanllal for aoodanta. ekparla tay. may be oompoundad by did fact diaf Loa Angalat ground 
controlars do nol always draci plott to tan lo tha ends of nanvoys balora taksig off. Piets of smalar 
planas. whose olicrafl ganaraly need leas distsnea to gal off the gmund dian larger tadnors. fraguantty are 
daarad for takaotf from naiway kaoraaebons. 


2007 Fsettva. Inc. Al nghls taaarvad. 
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H ¥»*s (nm tucti an nianaelien Ihti the daparene SkyWaM plana twnad onto Runway 24-LaA. wtwa t 
waa *tiXKk ^ USAif*i inoomkig Fight 1493. 

'That arpotl la known lor thia kintf 01 ptobiam.* said John GaipaiA praaldanl of Aviabon Salaly InabbJia fei 
Worffinglon. OWo. Tfa an a k tramaly tiuay aaport. maldno K vary (avoraMa lor aircraft to gel ou onto 
runways whara Svsy ahouldnl ba.* 

Last Apri. a rapon praparad by lha Aa Lina Riots Aaan. oor^cludad Ihc dia aaspaoa around Los Angolas 
Intamabonal Airport was among lha savan moat dangaroua light araas m Ihs Uniad Stalaa. 

Rials twva found that baing on lha groiatd can ba no lass Ihro ai ani n g. 

Tha NASA rapoita. oopisa of which wars racandy oMainad undar tha U.S. Fraadeoi of Information AcL 
show that ki January. 1966. a pasaangar jal baraty nsasad a smai phvan plana that waa awaiting takaolf 
iratnjcbona on 9ia trvashoid of Runway 2S4UgM. Tha jattnar^ vtsMI^ was hampamd by an sai^ morning 
haza. but It was an apparanl aw trsKc oonboltr’s anor dial pul both aWcraft on tha sama runway at dw 
samatinia. 

1 phonad dta LAX towsr and was told that lha igni plana apparanlN had boan daarod Into popbon (for 
takaofl). and than dwy (oontroHars) had l orgotlan him.' wrola ona of lha jalinar craw mambars in hla Inadant 
rtpoit 

In a Dacambar. 1967. WicidanL anothar dfoaslar was nanowty svarlad whan ona plot dsddad to antncata 
hknaalf iom taiowsy grUock-and staarad into tha path of an onoomng jal taking off. 

'Thara waia parhops alght to 10 jats Mhin a t.OOChfoal isdkis of us at this ima.* lha pilot raportad. It 
appaa^ wo wara In a grldfock with no placa to go axcapi a laodway M my laft whioi I baiavad to ba lha 
oular taaiwsy ' 

It was not a taidway but an acliva ninway. To maka mattsrs worsa. communicalion with ground ooriroiars 
was all but impossm baeauaa rado doquanelas wars swim pid. 

To halo har (tha eont r ofc r ) out with har astramaly buay workload. I took om Irtbadva to taka tha turn to dia 
lift laarang ma back to what I thauipit waa tha outar paialal but was in tact tha njnway.' lha piM raportad. 
'Slmitanaoualy. wa saw aWcraft tglas lacing us.* 

Tha piol hasidy bwnsd Into a groan Tarmac araa«aWy off dia nwiwaynu dia othar jalllnar roaiad Into dia 
nigMsky. 

Only a taw days lator. anothar past raportad taking off orar tha lai of a wfoa-body .at that ha drsoovarad 
was lunng out Into dia nwiway. Tha d isco vary was mads too law to abort tha takaoff. but as ha Htad off dw 
piol vaarad tall to Inoraasa Na daaranca. 

Ttad wa bat No. 2 angina or btoiwi a bra during takaoff rol. wo mighl havo struck mat alrcrafL' lha piol 
nolsd in his NA^ ripoiL ‘V our astlmation. dw towar should hara had dw wida-body tarri tarlhar off dw 
runway bafora daa n ng us tor taksoft.* 

Tha NASA tisa ara part of dw FAA’s salaly raporting syslam. Air Irattle c o n d ola r s and craw mambars tla 
tha rapoRs vckintarlly and. n so dolrig. oan rscaiva muinttf from adminwlraltva actions againat lhair 
Fadaral Avtation Adminisirabon carencaws. 

Tha Timas ravwwad scoras of NASA raports ralalsd to oparatton a at Los Angalas Mamabonal Airport and 
found numaroua axamplss of hazardous siluabons craalsd whan plots tardad serosa acirva nmways without 
oontrolar approval or mislakanly bwnad onto tha wrong tsiaways. 

In ona ISeSincidsnL a orsw got lost attoritwssdivartad toan untamaar partoftha airport to wasfor a 
plana parksd at Its ramp to dspart Wanding ttwir way back to tha ramp through a maza of blua isiiwsy 
ights. oaw mambars mads n laast two or thraa wrong turns unH lhay ancountarad whlw Ighls: Thsy wars 
on an scitva runway. 


Xraw knmadMaty . . . anpadltad taxi to trsi aroL* ttw plot wrots. 

Msanwhia. anothar fol dwt was descanding had to pul up abrupffy. misalng fw wayward arcratt by a scant 
200faaL 


2007 Facthra, Inc. Aff rights laatrvsd. 



126 


Exhibit ni 

factiva. 


Cos Angeles Slmes 


P;WT-A, MiSontJ OMk 

COLUMN ONE FAA'* SaMy RwponM Racord Htti Tii>1>ul«ne* Ovm th* past dacada, ttw agancy 
hat baan atow to haad tafaiy wamingt-toinatiinat acting only altar fatal crathat. according to a 
Tbnat ttudy. Sactaa; DANGEROUS DELAYS. FAA't Ratponta Racord Undar FIrt. FIrat In a tarlat 

JEFF BRAZIL 
TIMES STAFF WRITER 
3.1T9 wonit 
11 OaotniMr1994 
Los Angilas Tknaa 
Homo 
1 

Engkih 

(CopynghL Tha Tlmat llAnor Company; Lot Angalat Tanaa IW AI ftgMi Ratarvad) 

AtUSAv FEght 1493 praparad to touch do«m at Lot Angalat IntatnalionalAlipod on Fab. 1. 1991, 
pattangar David Richman. a Harvard-adueaiad cotaga prolattor and proud ntw lathar. know nothing about 
dia potanpal dangar ct rueway ooKilont. Nor did any of Via olhar 33 paopla who waro about to i9a that 
Friday avanng. 

But tha Fedaral Aviation Admailtiration did loi«v. 

At far back at 1983. rooordatliew, air tafttyatparli had urgad FAA action to do ntora to pravant runway 
acodanti. Evan wiViin tha agancy. offiicaMt had pioddad Ihtir MAianoii to aaparMa ground-baaod radar 

tyttama and ralativaly inaapantivt trallic Ughting tyttama. 

Tha FAA did taka twaa acbona and man-bit art/ altar Fight 1493 tuddanly Itndad atra a Skywaat 
commular plane, kflng 34 paopit. Thia wat tha Viird fatal naway coMaion in lha Unitad Statat in 13 
montht. 

tlow many crathat do you naad?' aakad Barry M. SwaacOar, diracaor of aafaty racommandationt lor Via 
National TranaporMlon Sa^ Board. If you have ona. Viare thould navar ba anothar ona with tha tame 
probfam Wry do wa naad two. thraa. tov or livaT' 

In dozant ol mataneat ever Vie patl decade, the FAA hat takan two yaart or moia to retpond to repeated 
wamingi of air taMyproblama. and Via agancy ollan hat acted only allar loat of ile. according to a 
four-month ttudy by uia Tmaa. 

Tha FAA. Via fadam agancy raapontibla by la* lor anaumg air tafety. hae baan named by tha NTSB at a 
cauta or factor in 103 airplana accidanit and nddanti batwaan I9e3andlatt July that lofiatharkiaad 574 
andiniured421. 

And raoordt alto thow that hundrada more people have itad m crathat cauaad by probiamt to which Via 
FAA had baan alartad but feted to act 

In addOon to runway coMalona. thaae problama nckidad twbulanca cautad by the wlnga of Boeing 757 
leOnart. prooadurea for da long wuigt before takaoll. the lefurbiahmenl of agaig airoafl pattangar tooau 
to tmargancy aalt and the IntMabor. of davioat Viat warn i a plana It flying too kM. 

Although ha taid a afaty it the agancy'a *No 1 fob.* David Hhaon, tha admmitlralor of the FA^ taid of the 
ktl of lipeet: *1 Viink you've pul your finger on t omeVi in g Viat wa'ra raaty Vymg lo deal with.* 

Wida Hiraon waa unfamkar wdh tha toacdic treat Tha Tanaa wat locueing on-mott of wtadi praoadtd fat 
arrivN ki I9934ia taid: Ttopafuby. wal ba more raaoonaibfe in quickar faahion.* 

FAA officialt tay tha agancYt day-to-day oparaion addraatat moat Ittuaa 'ralhar affectively.* cflan 
datacbng and ramadying tafety probiania bafore acodemi occur Each morning. Viay aay. the agency 
dapl^ about 3.000 mapactort and nait the buaietl airapace m via wertdwllhout mayor incidant on mott 
daya. 


2007 FaeVve. Inc. AN rlghti rtearvad. 
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Thu IS • fonnxMl* tatk.* Hinson isitf. 

The FAA 'sdoxwriedget that R can be raecOve In some ceeee. but that can be a pioper leaponae In many 
cases* acoonUng k> a ataMmenI by Ihs agancy In response to The Tunes. 

*Reaction o<lsn means we insist on laamlne (nun past aoodenea. At Mhar amss, reaction may mean that we 
are properly raaponpuig lo public demand or oonoann. which is proper in our damoorabc term of 
Bovemrnent* 

Said Hnaon: K you look at the way the FAA deals wtih safety, about 99% of what wa do ... of what you or 
I would cal presmpOire safety efforts, those ana efloits you probsbiy never read about or awlls about. 

‘About 5% of what we do is after there Is an Incident or an acadant* 

Airpisne manuiaclurars and alfimas said sslety comas first with both aie FAA and them, and that they would 
never compnunise the ivss of the flying public. 'We alweys want to try to make a good systom bettor.' said 
Joe Hopkint. spokesman lor United Artnaa. 

A schadiied commercial airplane top In tna United Slatas Is tsgarded ss amorw toe ss fe el forms of travel in 
toe world The odds of an air travetr dying in a crash are rougluy toe same as toe odds of being Idled by 
lightning. aooonSng to toe Habonal Sal^ CounciL 

The U.S airtne uidustry had gone mors than two years wftomt a mafor acddant before toe string of total 
accktorts this year. As of October, an esti mato d 843 peopla have died in plane crashes si f 994.ons of the 
worst records since 1968. according to preliminary stabancs by toe NTSB, the agency that kwasbostas 
acddants. 

Now. NTSB soiaoes say. nvastigators are auplorkto whether daltoeneie s In FAA oversight and operabons 
may have been cu nl r tt) u (UK| toctors to at toast two of toe three major aocidonls tois year n Charlotte. N.C.. 
Plasburtto. Fa., and near Oucago. 

AAsr toe FAA admittod in July that t may have miahandlad rspons on turbulenoe preblsm a asaocialad wito 
Boeing 797 jalllnare. The Tunes reviewad hundreds of aufirw orashee and thousands of government 
documents obtained under toe Freedom of Information Act. Scorae of peat and praeeni FAA offlclalt. 
members of Congrus. slrtne Industry souroes and safety eupeds were uitsrvlew^ 

What emsrged was a portrait of an agsncy tttot many bmee has bean slow to address safety problems, 
perbcularty whan toey were controversial or coaly to correcL Among toe Itodaigs' 

* The FAA's performance sometimes has besn compromised by poor oommunicstion between those 
charged with ideritifying potoraial safety problems and those with toe power to act on them, end p r e ssur e 
from an tnthianliai uidustry. 

'Any lime you want to change toe rules, toe manufacturars scream.’ said one FAA awaafl-csrtficabon 
oIBcial who spoka on condition of anonyrrUly. 'Anything toai cost money to the manufsclursrt. we have to 
fight them on. They say Wal, Siare haanT bean an aocidanl' We say: 'Well, there oould be.' * 

' Deacfy delays here occuned in pan because a law ractuues toe FAA to iusbfy toe cost of unplemsnting 
proposed safety meaeuree by shcwring toat anougn fives wii be saved. 

hfi strengiialion of safety by regutotcry pcocsae.* observed Rep. James L Obsrator (D4Ann.). chaaman of 
die Houae Public Works and Trsnsportsfion Commlltse's pansi on aviallon. 

* The FAA faied to heed repeated sdmonlUone fcom overaqhl agencies and from within its mm ranks to 
makt safety a higfisr priority. And toots exists, both inside and outskfs too FAA, t tacit aocaptanos that 
tomsbmss only aoddsras can spur Iht agency to taka meankigfvil action. 

Chsrlae 0. Mfler. farmer head of toe NTSB's Bureau of Air Safety, said he has kept a file calad htnown 

preoederUs.* 


T have baen keepuig It for awhie because I wee getting Murbad about eseing socidsMi happen from 
causae I had teen before.* he said. 

Considar toe case of runway safely. 


2007 Fsctive. ItK. All rights reserved. 
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nation^ bu«««t airportt hu Man batvnd lenadula and may nol ba oemplaiad unU tha ban of lha canbay. 
acoonfing is tha Ganaral Aooounttng CXtea. 

"Soma an n o ammam on tha badi bumar. and ( doaani gal done * said Hugh E. Walamian. a lonnar FAA 
managar who workad tat Bia agancy <o( 27 yaan baton laamg In 1M£. 'Buiman. onagovamor or<0 
ngular human baings i*aa. man i goaa onto tha from bianar * 


Oaatad at 18S8. Dia FAA haa a doubta-bacralad task that crIlSca baiava a a conltetlfig onr ta pccmola Oia 
aviabon industry and to ansika salafy. 

Tha NTS6 ■ chargad widi aivasligaang ma causaa of Iranagoftalion acodants and matang sataly 
neommandabons Craatad in t974.iihas no ngulalory powanand cannoftaroama FAAtaact 
tiiinrirstj tha NTS8 has iMeldsd ss tnllusnca moat suooaaataly by pubidy dranang allsraon to tha FAAt 
lapsaa and tavangkig lagislaem ouliaga. 

Tha FAA points out that through tha yaars K has idanelliad numanus sataly hazards and adsd baton 
acodants aval oceuirad. In a aH>aga stalamani ta Tha Tanas. FAA oflidais sad tie agency routnaly 
Oalacts sa^ problams but racaint no oadc because than a no way ta M how many aoddarts have 
been pravanad. 

'Aafna safs^ a among tha taast raoognlzad suooaas stanaa n pubtc or oorpor a ia poticy ta the United 
Statas,' tha FAA statamant said. Tmprovamants m angina ratability, aircratt design, avtaraca oocApil 
lachnology. navigational ads and tirtrsflic control have made saitoua acodants rare avants.* 

ARhough tha FAA has adoptad alghi o( aveiy 10 sataty racommandabont made by the KTS8. the avaiaga 
amoian of bma a takas ta implamant ma racommandabona is stghSy mon than two yaare. In soma casBS. 
rs baan much longer. 

WMta ha bahavas mat the NTS8 funeaons as a 'graat audRot* tar the FAA. Hmson sad die FAA somattmas 
reapaidUly daagraas with tha sataty board, and with good raaaon. 

1 thnk Vs probably OK ta have dasgraamanl on tha 1i% or so.* Hinton said, rafarring to tha numbar of 
sa^ recommendations tha agency doesn't aocapl Ift not |uat ... 'Wa doiTt nka them' and put them 
away. Wa have ta dafand our poslbcn. 

'Thara are often cases in aviabon whara knowledgsabit paopla can disagrae.* 

The NTSS's Swaadtar said die sataly board was plaasad wim the ^5% aocaplanca rata.* buL ha said, 
'diara are quila a few Important isauas m die other 1SH.* 


Raoordt show a paaam of delay In conacting a nunbar of probtama: 

' It look the FAA a«ia years ta act on a pioblam with Cessna carburalort after John R. Jamas, msnagar of 
the FAAta aircraft oartificalion ofltn In Alianla. had wnitan a memo warring: *CAUTION. Ihtre have baan 
rtpoits of power lost, savera in tome casaa.* 

Tha probtam was causing planas to stab and crash. During tha) sighhyaar parlod. at least 10 acoidanlt had 
oCGunad and had a dozen paopls had (bad. 

* A year balora die tap had of a Boang 737 nppad off over Hawab in 1968 because of malal tabmis. FAA 
resatrchar Thomas Swift virota a 77-page report saying tabgus In Amanca's aging air Itaal was a sataly itaic 
H la possibla tar a nunbar of cracks, each not sasiy kisaactad. ta suddenly join togathar and tarni a hxig 
cnbcal crack. ... A nunbar of llaats are curranily operating at doubla their aubaffy anbcipalad design Ma 
goats.’ 


•It look a decada of admen i ttons and at least tau major crashes batara tha FAA acknowtadgad that certain 
planas. such as early modal DC-9s and Fokkar F-28s. ware aspeoaSy sutcapbbla ta control pnabiamt wnh 
mmuM amouias off loa on dtalr wingt. 

Tsnyaart aftar the NTS8 had made safety racommendtsiont fodowmg the deaths of 76 paopit in a 1982 
Ak Florida soddanl n Washington, a similar dagady happened at La Guanba Airport In New York. That 
aoddant. Involving USAr Flight 406. kfftad 27. 


2007 FaeUva. Inc. Ad rights rasanrad. 



129 


In 1978. two rcMarchen, uwig stMUc* owopM by Hit Naboral Aaronaubcs and Siwoa Ad H i Wa i ra Bont 
Aviatian Sitaay Rapeating SyMam, itaued a raped that aaid Inomana by aircraft on tha runway* al 
eomrolad aaports rapreaant a a i gnlHcana aafaly Itazard.* 

Batwean 1978 and 1983. al leaat ttvae naar-coHaion* imolvlng mater teKnan oecurrad on US. naway*. 

In 1985. aftar two Norltiweal Aabrtaa DC-IOs war* aivoivad In wtial bi* NTS8 larmad a "potaiftialy 
diaaalrou*’ inoidanl In MinneapoHa. Ih* aalaly board urged 9i* TM lo aapadit* pretacts oealgnad b praverd 
c ofc wna. 


Per yean bte FAA had tucn pr^acti in •)* work*. Imdudkig lb* davelopmant ol ground-baaad radar 
tyaama to halp aa traffic ooniroliar* track akplan** on lb* tarmac and aanpi* igbbng tyatam* to anaur* that 
deparbng plan** do not (tray onto active naiwaya. atpadally dvamg atdameiiit waMbar. 

Evan wlibn lb* FAA. offioal* racogmzad 1ba( progre** had baen loo tkm. 

An inlamal FAA tnamorandum i*m ID air baffle manager* In 1988 acknovdadged lhal Via naiway Incuralon 
Ihraat raquvad hatbar acbon. It nolad that many FAA pia)acl* dedgnad lo deal wllb tie proMam wera 
moompMa and. aven H Inbbad. woiAd bar* guaebonabi* efficacy. 

That yair lb* NTS8 ralaaaad Iba raaidls ol a (pedal tnvaettgabon on naiway Ineurdon*. prolfanng to the 
FAA » racemmendaaon* deai9nad to pravani ibem. On* laeommendabon origmafty bad baen laauad 13 
year* earlier, but waa navar acted upon. 

During Iba naid lot yaar*. accartftng lo bitamal FAA dooumanii. fti* agency racabrod diapatcb** Irom 
many oomari ol lb* mduatry. urging II to taka pravanHv* maaauree. 

OcL Z7. 1987; John OBnan. dbador ol anglnaanng and air laMy lor tb* Air Lin* Pflol* Aaan.. aaid In a 
Mlar 10 FAA AsaodaM Admmiaaralor Anlb^ Brodartek; *S*v*ial aeddardi and kiGidania over di* bM law 
year* have dooumardad lb* aaiiouan*** ol nmray Incuniona.* 

Aug. 11. 1989. Anmtamal FAA mamoranthim dadarad' The Aft Traffic Oparat i on a Sarvic* ha* alalad an 
immadiala naad lor a nmway ineuralon alatt ayttam. . , . Wa agree Ibal tie potanbal lor a runway aoebent la 
a nabonal oonoam.’ 

But that coneam bjmad M eatatirepb*. 

The Km ol three lalal njnway kiouraion* ocourrad In Allanta on Jan. 18. 1990. lolowad by one In Dabdl on 
Dec 3. 1990. and than lb* Lo* Angale* craah In February. 1991. Tb* dealb tol for lb* thrae craabaa; 47 
peopl*. induding David Ricbman. via ooHege prof aaa n r. 

*Th* world changed at Ibal momanL* (aid Richman'* father. Alex, a psydaatriit and prolaaaor In Nova 
Sooba. “On Ibal... morning, wa war* ignoranl Wethougbtw* ooddbuet the alrln** and tbagovammanL 
W* thought (alaly cam* fm* 

After lb* Allanta ciaab wikeb hffied on* man. lb* FAA muatarad action taama to aunray aaporl* and identiiy 
potandal problem area*. It alao relaaaad a rapoil In ApdI. 1990. amobaalcng that pilot* could help out 
runway ncuriMn* by being mora awar* ol (unowfng airoralL 

At lb* Urn*, lb* agency alao acknowladgad; TAA baa not alway* ooordnaMd Ha allorta to reduce 
aiGuralon*.* 

StI. It wa* not unal aftar Vi* Loa Angale* craah that the agency placad mor* empbaaa and money Mo ila 
long-axtant mnwatHneurdon prp)*cL 

By than, aooordng to Via FAAa own aadmal*. 11 ninway acodenis bad ocourrad in Ih* LMlad State* alno* 
19^. raauibng >i 844 death* and mfuba*. not including Vi* eaauaftia* from th* AVanta. Lo* Angelaa and 
Datroft craanea. 

Tm nol a great fan ol bow faat we gat Wngs don* In thi* erganizabon.* aaid Vi* FAA'* Micbael J. Haniaon. 
who (pearbeaded the nmway coBmon prarmnoon program altar Via craah of USAIr FKgbt 1493. "Should lb* 
nmwaymeumon dull have baan don* aooner? Tb* anawar to Ibal ir of coma." 

Sinoa dial Vm*. Vie agane/a plan lo develop a aophcdicaled ground-baaad radar eydam and plao* ft al lb* 
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R wMrcMIiy JEFF BRAZIL UK) SHELBY GRAD/ Lot Ang«tMTitnM 
How trw Study Wn Condudad 

To compiM Mo rapotL T)* Timw rovwwod roughly 20.000 pdow 0 ) inMtnil dooumanta oMUnad from Vw 
Fodoral Aviobon Aduinialrakon Ovougn no FnMom of Intarmatieci Act Tht nnnpopor alao rovwwod a 
compuluanalyait o/diacauiato/airptarwcraihat batwaan 19S3 and lau Jtiy by tha Nabonal 
Tranaponabon Saf^ Board Tha Tanat alao tluddd dozant of rapods by govammant ovaraght agandaa. 
arM imanriawad aeoraa ol piaaani and paal FAA oflieiala. aalina aidutliy aoureaa. avMion aa/aiy aapana. 
mambaa o/Congraas. oath aurvivara arid via famliaa o/craah wcoma. 

PHOTO: COLOR. (OrangaCounlyEdlllon.At)ThlaUSAIrjathlttoanKTiuHr plana at LAXIn 1991. kilaig 
34 paopla.balona runway a^(a(ywatupgradad..PHOTa InvatHgatoia aurvay wracAaga of USAV Boaaig 
737VwmanKngaltu»iaFab. 1. 1991. daadlyooliaion wah a Skywaat Matroptana al LAX /MARSHA 
TREAGER GORMAN / Loa Angalaa Tma. FWTO: Alax Rennan. ftvm at Oawd Rictwnan. a wcdm ol 
Via rufway oraah of ajal In Loa Angalaa In 1991. aaya. V/a tioughl aaMy cama fnL' / JAMIE FRANCIS / 
For Tha Tmaa. PHOTO: (OrangaCooMyEiJBon. AM) WracAagaolUSAir ial at LAX. Many aboard diadol 
tmoAa nhaMon. S yaart aitu tha F/LA wat aortaad ol artiargancy-axit accaai proMam a / JIM 
MENDENHALL /Loa Angalaa Tlmat; PHOTO: Dayid HUiaorv FAA ad m lnl a t r alor, CHART Runway 
Inouraiona / Lot Angalaa Tnaa 

Documanl latmOOOOZOOl 1029dqcb02hlq 
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Th* AirUncPtotiAMn siM In • Mw to th* FAA « »• tinw: 'Pw lack o( an •daquUe rmpociM to (»» 
NTSBa aailar) taoonwnandaeon (on da-tono wMbo). viNon «•» mada appronmalaly 9 y«an ago. 
oonolbutad to ttw actidoM mvoMng USAIr FkgtK 40S.* 

TlwFAA.'iaid vataranariafatynnnllgaliv Rudolf Kaputlln.*mtkaalioundnia: 'Hay. wa'va got a 
Brand-naw preotam hara.' Wal. a lot of dmaa fa not a naw ptotilaffl. Ift an old proMam.* 

Kapuaan. wnohaawortcadtoraiaFAAandthaNTSS. waalhalaadinvoiligalorintha 1962 Air Florida 
oraari. wMeft waa cauaad. In part by ioa on lha winga. 'Attar lha FoKkar aceidant (at La Ouardia Airport In 
1992). Ilia FAA oonwanad dila wortdwida oonfaranca on dancing. Tnay aaid 'Noar yo/va got to pul a laddar 
up and loucn (•« wing toaeaVkltaaloaonlt).' Wad. Jaiua. wa knaw mat yaara ago.' 

Tlia FAA naaF ooncludad laatJuly tfial Ha aOfflly to aid upon aalaly conoama in a amaly mannar waa lacking. 
Tlw agam had launefiad an ntamal InvaadgaDon aflar Tha Timaa raporlad dial t had naaliandlad tha 
Bo^ 7S7 ItftxAanoa laaua. 

But fhe ardoiewtadgmanl cama aa no aiapnaa to air aalaly axparta. 

In 1986. lliaivFAA Adnunatralor Alan McArlor. in an lnlamalmanwrBndum.aclinowladgad that tha agarwy 
tackad an miainal daaringhouaa for aaialy data, and that managamant of aafaty data wAr lha agancy waa 
(taorganaad. 

A July. 1986. Offica of TacfinofoByAaaacamantraportooneiudad: 'Mora ttringanl aafaty atandarda uaualy 
fdow a widaty pubUdzad akkna accidani and vocal public and oongraaaaonaf ooncam than from FAA 
rnibativaa' 

And laal yaar. RoOart E. Macfid. Ha FAA'a cfilof aclanoai bafora rabring last aummar. poksad out tha 
aganc/a ahorteomnga. In a Mardi. 1983. mamo. ha datalad hew ttia Boaing 757 waka-turtulanoa problam 
waa hamlad. Bafora two craahaa that daanad 13livaa. MaclialhadpiBdcladlhaiturtiulancaersaladby9ia 
jadkiara would cauaa a 'nador oraafr If tha FAA lallad to taka pravanbva maaauraa. 

'TNa la aymptomaibc of a bogar problarn. which la that wa raaol vary alowfy to thinga whara mayba wa 
ought 10 aot nwra rapl^.' Maidiol wrola. 'Wa naad to ba mora atari to aafaty guaabona.' 

Naxb Why dalaya In rsaponaa ooour. 

Tknas raaaarehir Shada A Kam and oorraapondant Shalby (3rad con trt b u tad to Wa raport 
Tha FAA Factor in Craahaa 

Tha FadaiM Amabon Admmat r abon s latad aa a cauaa or factor In 93 plans craahaa lrom1963 to July. 
1994. Infury and fatality raoorda tor dia 93 craahaa ahow. 

*57edaad 

' tSI aanouafy Wurad 

* 270 modaralaly Inlurad 

An addWonal 10 craahaa wars atbibulsd to arrora by ipacSe FAA amployaaa. raauWng In: 

'Savandaad 

• Bgfit modar a ialy injurad 

Sourea: NaBonai Tranaportabon Satoty Board 

RaaaarchsdbyJEFF BRAZIL and SHELBY QRAO / Loa Angalaa TWiaa 

Ru nway Incuaions 

From 199010 1993. Ihamimbar of aircrafi runway coKaiont has dadnsd by ona-Wrddaapaa an Incraaaa 
In aa traSc: '93' 168 Sourea Fadaral Avisllan Admlntal r alion 
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ABSTRACT 

MWtln NASA * Aymcn SalMy Program. »m Ambon 
$lislam Monaoring »id MocMng (ASMM) Pro|act 
mMtmom tha AMd 10 provida 4aaaion fMliM mtft tha 
tooto to ktonofy and avatotoa pradtoponng oondBona 
toat ooiid toad to aectoanta TMa Protaei to da«atoping 
a aat or autonwaad tooto lo laaMtoo aAcionL 
oomotahanawa. ar>d aocuraia analyaaa oiT data coiactod 
In larga. hataroganaoua databaiaa throughout the 
NaOortal Avtotlon SyMam. Thto raport to a brtoT OMarvtow 
of lha ASfcdd fV otoct aa an Mroducaon to iha raatoTOta 
praaaniaOona m into taaaton on o<»a cT at Itoy atowann 
•lha Parfonnonaa Data AMlitato and Raport to g S ya tom 
(POARS) 

IHTROOUCnON 

Air tramportaaon to imram to oonanuad aconem c 
dawatoprWi of toa amid AAhough • to ona of tha 
aataat modaa of travai too puliic damanda toat tatoty 
tovato conOnuouafy totpr o wa and that fia abaotota 
numbar of a%toOon acddartoa oortonwa to dadna. awan 
aa ar iratoc tovato inoraaaa Thata to a raoogrttoad naad 
twoughoul tha mtomattonal anabon oommunihr to 
bocome avan mom proactNa In managing aalaty rtok aa 
a««dancad by via toiowang aiatomani maoa by (ha FAA 
Ad mwb atof. Uanon Blahafy at tha North Amanean 
SatotyConfarartoaototoartNayaarTorona. aanaadto 
ohanga on# of lha bipgaar httfoncaf oharacfanaOca of 
oktobon atotofy laifvottomann oir raacdva nitoira. 
toto aiuir gar in bonf at acadanto. .onbapoto tham...and 
i#oa ftoni dtou to daaacf prototoma and ddrurtirig oanda * 

A proBOtora ap proach to Mardfylno arto abatoaOng Ma> 
thraataning condtoona invoNaa monaoring Vw ayatom 
parlormanoa In a nort^puntoya anvironmanl. to ar r mg 
from normal opa raO onal axpartonc a . idanllfyvig lha 
practfaora that ioraahadow moat aocidanto. and 
Oaaignng appropnato intaiyinaona to iw n a nua tha ntot 
of Vwr oocurronoa. Dactoion-makara muat ba abto to 
fbeua giAetoy on vioao avanta wm via hytaat potontw 
aarorty and toa ihood of rtooccurranoa 

Tha govarnmanto and tha world ai ia bon oamroundy 
rouAnafy amaaa torga guorwaa of data tfiat eouto ba 
aourcaa of mfortnotton rotovard to a iaaio n atototy. 


Irving C. Sltotler and DavM A. Maluf 

NASA Amaa RaaaartA Cantor. Moflba Faks. CA 


tocraaaingly. lha accumutolton of thaaa data outpacaa 
toa comminto^a abdty to put lham to practic a l uaa. 
Ofton aalaty data eannoC ba ralriavad aftor lhay havt 
baan put atoo oomputoftted atoraga baoauaa of Vie way 
ftat tw data wara catagoruiad. K to MBcub to oombina 
data ratolad to tha aama aubfacl whan thay ooma from 
tWaraa. hat ar oganaoua aourcaa. Tha abdVy to mcndor 
eonanuoutly. convart tha cobactod data into rababia 
m ltann aaoiv and itoart that idormatton tor coiaboraihra 
daciaion making to tha baiia tor a proactva approach to 
idardfying and aBevt et mg Wto thraataning adatton 
oontobona and avanta 

THE AV1AT10M SYSTEM MONITORMO AND 
MOOELINC PROJECT 

Tha Adabon Syaiam Momienng and Modabng {A8MM| 
proiact of NA^'a A^bon Satoty Program, add r a aa aa 
Via naad to prodda daciaion makaro w<h tooto to aaatot 
Viom in toanflf)^ and oonacting lha pradapoatog 
eondBonaVtotooutotoadtoaobdanta (Raf 4) 

ASMM doaa not am to raptac a himan aaparVaa with 
automaaon Rathar. b proddaa eompuialiorai tooto to 
irananBa damanda on iHman oxparto and to tocua Viair 
attonbon on vio moot aigrvAcont avanta. and hato Viam 
tdarafy Via toctora that dabnguah unaala oparabona 
bom roudna bighta. n haa davatopad tooto to do tasba 
Viat ptatanby can only ba partormod wfh much torn and 
adtort by adabon aiparto Tha ASMM tocto oonvart a 
bounty cf raw adabon data drmm from marry aourcaa— 
aircraft flighl data raoordara. ATC radar tradto. 
malniananoa toga, w aa Vw r raoonto. aitoiton aatocy 
mcidant taporta, a to tito meanirn^ kifannabon. tortfy 
daptoyad Tha tocua of tha ASMM profact is on 
al ari tf |i1n g praciaaor condtoona that atovaka tha 
probabtoty of downa v aam human arrora Viat may. in 
tom. c oniribma to aywtoon a a tocy Inddania or aotidani a . 

Sad) of tha aavarai ASMM tooto oonbtoutoa to a unkyua 
raight mao Via comptota pictora of a aalaty avam. and 
can ba uaad to auppM a oomptomanCary and aynargiaitie 
piocaaa of cauaal analyato and aalaty rtak aaaaaaroant 
from a ayalam wtda p aro pacQva QuaMabva data 
aouroM yiaki i nto rmabon Va4 hatpa tha anaiyat 
undaraand Via aubfaovva aapacia of *why* an ncidant 
ooourrad. wfito quanUabva data aourcaa haip tha 
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to k^tdorftantf 9io ob|oclh« npect* ol Vrftat* 
hapiwiod. 

MOfffTOR TO IDOfTIFY SAFETY-RElATED EVENTS 

Th§ fM Mp In tfto proocth« monogtmtn* of iNic N to 
fnonitor 9m lyrtwn oontnuouily. vd ooloct. codify, and 
daaaify tafacy mddafK data Mo ftpoMorfat tfwt can bo 
aubaaohianey inM tor aafaty tniigNa Tfio databatoi 
oontaMno mtormaoon rolavoni to avMtton aaloty ara 
vary lofga, hataroganaoua (ladual and dtoMO. dfvaraa. 
dtoPtouitd louroaa from aWcti miormoaon muat bo 
aalracaMt and margad to gan a oomptala ptotiaa of a 
aftiatton. Ttw Mormadon muat ba daplayod In a way 
that maMa I aaay tor toa domato aapoft to Marpraf and 
to oorapara wtoi a a pa ct atona or parfarmanoa 
alandarda., and to gain tia Inagfil naadad to idanlify 
Ihoaa avanta tfwl praaant polanflai rtalca 

Soma of 9m dataOim> aueh aa tha Avtooon Saiaty 
Raporang Cyalam (ASSS) arto Nattonal Tranaportadon 
Satoty Board (NTSB) datobaiaa daal wNh lha nadonal 
avtobon ayaianv Odtora arcMva data app fl oabto to 
pardcutor groopa of uaora. Aecordngfy. toaASMM uaaa 
a dual m ortoo^ abalagy. M davatopa toola trial halp 
idanOfy tyatom wtda aaito^ iranda vmitQ atoating and 
avoMng ayatam toval daCa raaounaa (aiownural 
moratortogX and a provtdaa nMdual oonatttiianta of lha 
NAS aritfi toola M anabto tiam to draw uaalui 
intormadon from tha data thay gadiar (Mramuraf 
momtortog) 

IntrinHit MBfitennfl 

Tha tottonaural Afondoring alamant la totondad to prortda 
tha ad aarvtoa oparalora wtot toa tooto naadad to 
mondor thair own partormanoa and aafaty condnuouaty. 
aftOctvafy. and aoonomcaty wViin thair own 
organizadona Tha primary producto of tola aotodty ara 
tha AnaMn Partormanoa Maamatng Syitom (APMS) for 
prooaating HgM'iacordad data and 9ta Partormanoa 
Data Antfyiia and Raporang Syaiam (POARS) tor 
pfocaaaing aa iraMc con t rol data Tha tnlant ta to 
prowda a aulta of tooia tor comrarting daAa Me 
Mormadon cuitomizad to tha naada of each individual 
tiaar. and. tharaby. to ancrhaaga toam to ahara thaw 
I ni onnaeon lor oooparadve proactria dadaton making. 

Irtoamirto monaonng al tha aa camara la addraaaad 
wdh Pia APIfS dial la tha raaaarc h to dmmlop toa 
matoodotogat and toola to damonciraaa to US aw 
camara that vanr larga quanosaa of llgtt*raaordad daaa 
can ba mon do rad. procaaaad. and anaiyxad routtnaiy. 
itociandy. aconomt ca iy. and maMy, Tha aiito of 
kitograto d APMS toola la daatgnad to oonvart 
raoordad date Mo Wdorm a tton to Via ak^aamcaa 
provtdar tor aanatng tha Quatty. rahabMy. and aatocy of 
partormanoa of aaoh oocrpaiya own Fb^hl Opara d ona 
and QuMty Aaaumnoa (F<XM) pro gr ama and Adranoad 
OuaiAcabona Programa (AQP) (Raf 1 and 2) APMS 
toola go aubatandafry bayond 9m fapatiamaa of Via 
owTom commarciaiy tofr wa ra programa Viat 


ara mairWy daaigr ud to count pra dainad a ao aa dm caa- 
Tha AMAS wdi aaawt an oparitor in undaraSandtog how 
aa akoaft ara baing oparatad normaVy and roudnoiy on 
thalna. Tha fight-aafaty anaiyat hM ba abto to idantify 
atyploal attOaacaOy aU rama. and aalWiNatBlad avanta 
and Vanda to luppM aafaty and aoonomic dadalona. 

intrarmwal monitofrg af air trame oenroi la addraaaad 
wkh tha NA8A-FAA Partorm a noa Data Analyaia and 
Raporvng SyMam (POARS). POARS la an ATC radar* 
track monkai^ c a pabt y davaicpad by NASA and lha 
FAA that roudnaiy eodacta. pro c aa a aa. and margaa ATC 
data, oompvtoa quanvtativa p artor ma noa maaaurat: 
produoaa and daaaminatoa daly partormanoa* 
meaaiwament raporto. ai>d ar cMvaa baac oparalcnal 
data arto partormanoo a ta f iad ca POARS partormanoa 
maaauramanli ratoto to ayatam VirougAput dotaya. 
ayatam pradictaMly. and othar kay ATC partormanoa 
todiealora. (Raf 3) Thia profad la baing oarrtad oM to 
colaborgaicin arflh Via ATC oommunty (FAA and 
NATCA) to Obtain Via uaara’ avatuationt and tha 
toantfadMormadon ai naada of aVVtoBcmanagamant 

Currartoy. Via ATC toaUbaa to thiat of Via nwia FAA' 
ATC ragiona piua Via Command Cortroi Cantor ara 
padiripii^ to tha toot and arvaiuaOon of POARS Thia 
conaiaulaa abcul vwty f ac to b aa oormaciid to tha 
POARS natwork and laoaiving raporto aadi morning 
about Via pravioua da/a oparabona Viat ara oiatoanizad 
to Via naada of each tocRty. By agraomam among tha 
taoBbaa, thaaa raporta ara aharad 

NASA to raaponatoto for Via im ptoman tab on and 
makitonanoa of tha aacura. dadc a aa d natwork ovar 
wWcfi POARS raporto ara dtotrtoutad and aharad among 
(aotobaa- Tha POARS natwork promto a for oobaotoig 
data from awch ATC aba. Iranamil^ Vwm to Via oantral 
ftoa tor pfocaaaing. and dabvanng Via raaiA of Via 
procaaaad data to ATC nwnagara at aacti of tha aiaa tor 
avakiabon (Figure 1). 

Tha funebonaf raquMmanta of tha FAA cuaiomara on 
PO^Snat toeua on via naad to maintow a aaoura and 
raiabia ptofi to aacti of Via data caraara, whila 
maintaining Via Sajdbdity for totura upgradaa and 
a ddborml aiaa POARSnat indudaa a two*barad 
approac h tha phyaicaldogtcal oomactMly batwaan ailat 
and tha aaourty madiantoim raqurad by Via proprtaiary 
naturaofViaATCdaaa. Tha PDARS wida ama natwork 
(WAN) oonnacthfly raquwamarka ara mat wah Ctooo 
2$24/3S21 routora at aadi tocadon on mufbpto Varna 
ralay ctoud. Tha Frama Ralay tac hn ology to a rakabto 
ooat-afiacbva aoMion that atoo oftort tha banabi of 
togtoai poM-topoM connacOvtty and bandwtolh 
upgradaa without 9m naad to natal additional 
aq U pmaM POARSnat haa a Commitlad Vitormabon 
Rata (CiR) of 3S4 tolobytoatoaoond ba t waan r ameda 
taaa and Via canirai prtxwtamg ala. gu a ra n laea 
availabMy of aarvtaa at 908%. and mapdrmim bma to 
raaiora aanfea to no mora than 4 fKAM 
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S«cufty H » pfVn* oonotm lor tMo nolwoilL Thtroltoro. 
mony prMMtom tn toh#fl lo «nohM dato 
oo rTto onBoHy. Thwo induit* Ih* pf»y«leal arid l09ieol 
•oMod of thia oalwork from o0«r wNmrt n 
(including ttw MomoO. oontrai monooomoot el WAN 
tocumy polOM and prooodurm. Mid ifrlci •nferoMnont 
of aooM from tte IAN ID PDARSnot rooouiOM. Oato 
•ow to Mrfifoolid lo Mciaty Mm tud < 1 ) am 
vfipiamamad on frto laal itoa roulM ( 2 ) opmwm on 
aouraa and dMdnatton addra ia a a . and ( 3 ) aei aa aca a a i 
Itia le adow only appmvad cuatomor notaoras lo 
tsvarae tha POARSnai 

Al met* FAA tola. NASA prcMdaa, opar a laa. and 
maatotona tfra nJtm lo lap »do f>a PQARSnat Ona 
LAN aooaaa port to pco«lM lor aacfr laaf oaa. Tba 
routor paaaaord to roMrleaad ID POARS opamOona- Tha 
damarc a Bon of raapontofrtofy lor t>a POARSnaf to at tia 
lAM pod on Via molar al tha tooal ada. FAA to 
raaponatia for ad raaoluDona that oxiand bayond frid 
damareaOon atoMn tha ada. 

Tha PDM^Snat to curtortoy a foi producBon notwortc 
ocmadlng about ffurty adaa and to cqponAng 

Eilramufto M oQtoyiM 

Tha Eiframuraf Mototortig ato m ant oompiamanm 
Mranaunto Mondorvig and pmvidaa a c ompritoia oa M 
macftantom lor montoofisQ t>a p arlormanoa and aalaty of 
tha ovaral National Aitodo n Syalam and lor data cBog 
and avaAtoUng tha aflacta of na«v lacftnoio gtoa aa ihay 
M inaadad Mo tha ayiaam. Extramurto hiorMOfing to 
Via *totodoMifi* alam a nt of Vto ditol atralagy lor 
mondortng Tha primary product of Vila acMty to tha 
National Am aPo n S/itom Opartotonaf Montoorin g Sannoa 
(NAOMS) 

NAOfrlS to a oomprahoraM and oohamnt aurmy cf tha 
oparaaora of tha atoodon aytoam (La., da pdota. 
Dana Qian, machanica. dtopa lc fiara. togfil aoandartoa. 
and otoiara) on a ragular batoa. Tham to provan valua n 
vtonmiQ tha avMfron t yatom Wough tha ayaa of da 
opar a aora. NAOMS la a longtudinto navay that wfl 
track aafaty tronda montoor tha impact of tachnotogicai 
and prooadixto dtonpaa to tha NAS, and oorartwla to 
toa dawatopmant of a dMtodttvan baato for atoaty 
dadtoona. 

Tha oonoapla and capabitoaa of via too approacfiaa 
(la. top-doarn aodrarmral monPoring and bctoom-up 
Mramuraf monaonng) ha%a av c Nad tidapondanffy In 
paraPto. Howavar, MbnnaOon dadmd from aacfi wdi 
oomplamara Via othar aa wal aa Via othar ato m a nta of 
ASMM in Vto procail of Idanafyino pracMora. 
monaortng vw aflaato of changaa. and da v alopi n g 
pradciM capaMtty 

EVALUATE THE OPERATIONAL 8 IOMFICANCE 

Tha aeoond atop to tha cyda of proaetfra monagamant 
of rtok to to aiitouato tha oparao ^ ai gnd ca noa of Via 


toddanl or avara that wm idan u ftod. Oactoton-makara 
mutt ba abla to locito puictoy on Vmaa ovarea iMh tha 
h ig h a at potantial tor aav ar a co na aqjancaa and thatoiood 
of faocc ir ranc a . Thto vrahjoann raquna an 
undoratandng of Via oonlaiduat factora and co ndWtona 
that atom oon du dm to Via t d a nt toa d incident to Vwt tha 
domain aaport can aaoartain tha I M to i ood of Mua 
oecunanoaa and aaaaaa Via aavarty of potonlito 
oonaaquancao 

Tha atomant of Vto ASMM Proiaei eaOad OiW Ana/yato 
Tooto Poitoiopmanf to Oavaioping a aat of automatod 
looto to facMala afnctort oompmhanaNa. and aooMta 
anafyaai of data ooBactod from torga, h atoro gen aoua 
data aouroaa Vvoughout Vw Nattonat Avtotoon Syalam. 
Thaaa naw tadwotogwa axtract tdnrmainn from and 
oaaibton matoWigful Knkagaa among boVt puaBtstfrt 
Oto.. taidual) and quMtaafrm (La., dlgitof) ilatahaiaa. 
and pRMda vtouaMationa of tignAeani pottoma and 
Vanda 

Vifonriaaen rmiat ba axlraciad from pua toa frw data 
aoiaoaa to hoM tha domain axpad imdartoand tha 
aubfactha aapa^ of ’aifiy* an toddam oocurrad. and 
from towndtawa data aouroaa to undamtond Vw 
obfaetha atotocto of \ihar happanad Tharatora. 
■tvimitotl capaMWaa am batog davatopod to p r o o a aa 
both taatoai and mananc avtodon data, and to aodracl 
roia¥ani tofamadon from duoraa d at abaa aa : todudng 
thoaa darhad from Vw adMbaa undar todtomumf and 
Eivamural Afondorihg Tha raatRa of Vw aaarctwa of 
h a to roge n ao ua databaaaa ara p raaa nia d In dtoptoyt of 
nwodngtol mlomwaon Vwl hoip tha analyita achtona 
tha t na rgN naadad to uidarattnd tha cveumatanoae. 
tooua Mr aa anaon on oporabonaty togrricaot o%anii, 
and propoaa mdigaang aedorw. 

Each of tha tooto demiopad am being laatod and 
tW ba to d by our partnom to tw opamionai anwonmanf 
undar Indamumf and Exframurto Uontomg. 

Tha toOdi being eanlad out undar vw atomant caOad 
Itodafng ma Sanutodona to daacrtoed In Vw next 
eacdOA at ■ ratoiaa to tha IbmiJaaon of an Menwn tl on 
HoMWvar, toaitinrw tlmutottona am atoo Mod to 
Evaluala an idaniflad ovant by haiptog Vw analyit 
oxptom lor da ooMaxtoal todora Vwt am oonduchw to 
eatore and iKman arror. gam awight Mo Vw oparadonal 
ai gwii oa noB of Vw avanL and aiiaat ito poitrtoal 
oonaaQuancaa. 

FORMULATE AN INTERVENTION 

Htoring i d artofla d an oparaldnaay aignMeant avani and 
<, r »da rato od aa oontooduaf flactori. Vw rwxi atop n Vw 
prooaaa of proacflva managamani of rtod to 10 tormutoio 
■nMarvanOon. K to up lo Vw eomarto In indualry and Vw 
FAA to Fonmitota and lo ImplemenC Vw Marvaniorw. 
Hcmavar. an obfadtoe of Vw ato m a nt of ASMM ca li d 
ModaMg arto Sfrnutodbrw la to aid the daoaion makM 
VI vwea toro toapa of Vw proeaaa. 
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ModMIrip antf SimUMm um moM or rw ^ 

M Of Mai •uKeiftnt 10 kay MiMy cAArMrtM 

lor roloblo pradkaon of tho lyoliinwwiOo oiool i of now 
tMfmolooiM and procoduraa on oparaOona and 
oowwTHini ca iiona Me dak tnoorporala human 
portormonca No oioMIno NAS rnodOIno tooli and ara 
being vaidaiad wm data oMarod from inpamunaf and 
EalramorM Mondonrig Technlqijoa have bean 
da Moped for raproaonOng rmilb-opaNlora NarMfog ki 
compko tfyrwntc aoanadoa 

FaotMie sNulattono aerva aa a computaoonal laal bed 
for arwiyzng tyaiam partor ma n c a. foefudfog foe 
coNrfoutforw of ir^Ntdoal oparafora. indMdual aiame nt a 
of foe ayaiam. foe miaracbono among rmdQpla aganta. 
tatfmoiooiaa. and large acala ayiM bow and oonooi 
wauaa. Faal>Uma ay men aide lamdahoni onaMa foe 
aaiafir anafyat to anowar ggaHeni Boa Ttoaa foe 
eofoPon havw any eecondaiy, prctoagafod or aide 
adbotaV and ’Doaa foa aofobon proNto for graoafor 
degradMon to unanatoparad operaadn a n o n aato t ^ and 
t>oa« foe propoeed MarverMfon addw ae foe rtgfn 
oueedon and to foe dghr way baaed on an undanfonotog 
of foe >oeir cognAN aysfono ^ 

The aaeaaefnant of aaAety rWt la curronty a pool hoe 
antoyak by foa human aaipart of foa MaOMKai raauba of 
foe Caat'Ome Monie Carlo eNaaNone Howmmr. 
analylical tooto are bang developed to paraM wlfo foa 
loe(-dma atoiulMforto to aaaN foe analyeto in toantiywg 
foa algntfcanl contortual laclort of an avant and to 
aaaeeimg foa aalaty haka. 

IMPLEMENT THE INTERVENTION 

krpiamenlaOon of an Naciertoon for an toeniBad 
probiam la aoeompiahad via protofypaa. foair 
effacavenaao la evaluatod. rednamarMi am 
enpiermrtiad. and foan fol acala daptoymanta ara 

licttotoed 

Tha atop foal to oAan musing from foa cyda of proacfoe 
managamani of rtok to foot of having to plaoa a tyetom 
tor monaorlng to ordar to aaaaaa foa aflactrvanaaa of foa 
mtorvanOon maaaurad agamn aepacubona. Thto la 
comparabto to foa INI atop to foa proecfoe- 
managamani procaaa calad Monitor lo fdanttfy and 
doaaa foa loop on foa qrcle Thto atop radAraa foal 
fooaa data thal are naeded to tveluato foe Nerveraon 
am appropnoiMy ooBaclad. oodited. and daaedlad tor 
rafroepacave aaarch. Tha monitorinQ ayatom ahoiAj 
have been to placa bttom foa totorvan b on lo gatfoar foa 
baaaina data for companion of foa twfom to foa oftor 
Onoa again, foa mlavant tofo r ma dcn in iarga 
halaroganaoua>, f ftt f'foirtrd databMai naad to ba 
margad to gain a com ptole piettfe of foa ayatom wtoa 
etuodon AJI of foa ASMM tooto am appleabto to 
(aaMMa Mdant and rtoghdul onalyaoa of al ratovam 
Inlormidon 


The abtoly to monior oondnuoutoy. convert foe cdec to d 
data Mo ntatoi nformaCion. and aham foal toformatton 
among foa atafcahoidira lor oNa b ora lh e dactoion 
iMtong to foa baoli for a mvotottonary. oroacfrim 
approach to mana ging foa avtatfon ayatom tor 
pmvanoon of aoddania. 

Tha four aub atomanta of ASMM (Cdramuraf MoAfodng 
foframiNf MoMortog. Oala Anaryato Toofi 
OaMopmanc and Medaing and Sanutooona) am 
totordapandam and intorrelatod ASMM wto margt foa 
produeto of foaea four ato m an i a into a aysterrvwlda 
frema worli anobing cd a boraaon to aviabon aafaty-rtak 
mana gamani by poley mahtora whafoar foay are to 
govamma n i or mduafry by aharing mfonnatton efola 
feapaobng foa prcpr to tory righto to aomaaOMToat of data 
and aanaOvtoaa to polNOal mtowaa ahouid foay ba 
ratoaiad outotoa foe owning orgwtoabon. 

Eaifo of •« ASMM Produeto aueh at APIMS. ROARS, 
and NAOMS hot alandMona capabtohaa fool wll 
oononua to avoNe aa foa Oala Analyato Tooto ara 
adaptod to maal foa avoNtog naada of foa 
co n wauancto a However, tho eua and ovar r ldinQ valua 
of foa ASMM Produeto to aa an Ma gr aaad auto of tooto 
to enabla foa a ctoavamarM of a ay a to m wida parapack 
on proodhe managamani of foa aadoty rWk of tha NAS. 
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Human Systems © 

integration division 

National Aviation Operational 
Monitoring Service (NAOMS) 
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ASMM «»«•• pINpOM 
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prtdMwey 

t DamoneMMtttuetorMi 
m^cfMnMn V 9 MM« Iwidt In 
NAS Mli»r and 10 «f»na»r 
tadOfi dnitfig tOM tanda. 



S. •**ciancy tfNci* or nv« ftgM and Air Tr^ Mwiagamar< (ATV} 
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p ortoiwnoo iPd M ioN or iht ovofod MS md tor dotocang mc e»o*uir>Q »» oftocto or n— 
toc lw otegtoiof trocoJi^eoMnoyoro lwoortidlnlBtNoNWlow A now oonoSAianey 
4 eomnofcM ligni crowt. OA pMi^ Ate cororolon, lodviieMrn. tlgtd oBondvdo. old) II 
oddodtolhon/voyochy— oo«iiffipouplDro |VM Off< rtrrd tom id d tw l U dioh O Mwi- 


NAOMS pwMdw on attiOy Id fcjppcrt tw owaBo o oowowundy m m lO M Oomon r of 
opOfiiloft^ogltNdP»Odndcilt^doWcocyolQOvo»nw»P1nd>j|dyOdofvo no on*. Tho 
NAOMS Toom two. fftoratoro. ouvwMod doo* aaoocMlono MPt ropioo< nio »<oo or if of 
dw mooNjidiri In vw iwiMdn conwnunf|r. 


NAOMS rw dovOMd I OTMl ^ dT morgy 10 dovotopoig ■ motiodolo^coly oound 
■urvoy proeaoo. Trado off tiovo boon oonodarod anonQ procMon. oocMdoy. ind COfi 
Tho fmm vmiblo ttai cart bo mmpuniid 10 aceorapliri fwao aadooiN la oampio tta. 
A vary auoeaiaMM Trial at NAOMS mFYW^holpad to quwitfytvoobadM. H 
tooo halpod to aotMUM) aovam otwr Noluraa of tia fnadtodotogy to orouro atoOMry and 
wtorffotoDM^ oHhoi»aM>cai»r»di. Ad^onoodWtooioiimoftodoarouaizodto 
proowo too dam and adraa tho ntonwalion nAo m Moaty 
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NAOMS Purpose 


To fill an aviation safety data gap 
through the collection of primary 
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R&D ACTIVITIES 


Information Analysis and Sharing 

an m«n«i»mtf4 lyMttm to Mf>« 

M 0t« tounflaUofi fpr ttto of Oato trandt and 

eh« idafiuikeattoA or pMamiad rtoxard* bofora 
acektoAU oecwr. 


Safety evaluation 

Davctofi ma U tod and matdci to maoMira progrdto hi 
faductof to« rtto of fdUfWM and ^fNAcant 
by two^Mrda.** 

{AMitMn Saltoy toA AaaM 


**'****• *— * r~T [ rn‘Tmi m rri u n m ar r nn i rfi md 

f o m mil toidbcatmh p rtimfmitaaamamadratatf 
fraup* UrCOOliM)^ /(jmiAnMMS^fUto 

3013: ViMitf th« Nft iiMbM C^— o>m (KI0> AMbiMc* 

nwap— f o f wm ap c tot th*>iwrm«f o>wa i itoi T 

■ d ir wun i o smaag JPOO mtmkm npiciw UVDO Of 
MO) / (AvttfMi SofKy AMlF««r* 

Mil: Cf p]«io tbo KCATS AotttMw Sdoty EatofmoOM 
Shoma (A51A5 ) Phot I ew-wl—io - 
ttOB ortanbM. miAadtac coocopt do^wawi. OFOO 01 

m I 

OoMlop ■ tyftom to incrooM wfoty of oommorctol 
oporottona. 

3011 PtP tk p otf tom oBM toob to lototot i t i nb oi w ht p» 
tcBiul Mfory btuM. (AvUeleo Sifocr liitk AfuIrMJ 

3013 DtBOMOMi « moriflf pwwyp t af narwwli boMd 
ttUMpuMo af mlamaOoB aniit^ fron Awma. da- 
mbutWamfoa (AmiMo SafwylUto Milyi) 

Safety Management System 

Produeo Qtodabn aa lor dtvotopim preoa a aaa ond 
tocfiooia^i to totototnani • aotoly monadamoot 
•yotom. 

30U: Oovdep pnef of coocapc for NoeCoti tnfM l n d a 
pfottcypa la oapicnwRi on a trial baoa vMli ariacta£ 
parisdpastt that Ipaafu a orooi lactian at m tanka 
p wtoM UPDO Of *71. 73. 73) / (Ariariao Saftoy 
toahAMlyfila)*^ 

3011 Caoytoti aoidy af nto baaad fUtt matuptiMat Car 
■ufl-aMvlaoa conootiad apaatlaoal aaiKy UPDO Of 
•dl]<A4li«AMayO 

3013; Doailcf rito ouAipmaM ca n ap ca . madak. mi toeCi 
far wammmi avoafi irram UPDO 01 aOO (Uo' 
BUMwd Attnafc $T«Maa laMardO 
3013: Dmw>«p riak wnwyiiiaiu caacaftK iwnJaft. and taafi 
fartaaaapertcatarayarplaM UPDO OC a«A (Ana' 

Oco 3afc^ lUalt Aaatpm) 

3014 OaBonamaa a Nifiaaa* L«««l S ya la m Safttr Aaaaw 
■MMt capahOit that «dl piwari^iy riattfif acaarf* 

rMfc aoaaa thr NofCaa UPDO Of 071. 73. 73}/ 
UPDO Aaaaoae Safity iMk Analyari}*^ 


301b f^trrmiTnTT i frmrltrtrf fTtiirrtTrn In Thr tut ff falii 
baa and tefariaa- 

2013'. Dananalnta a ana hiif ladurtian aa tha lata of (aub- 
Oaa and trpataa 

301S. Drnwnitraia a fa thirdi l ada rt to n ta tot nta af to- 
talDa and aduftea 

Capacity evaluation 

Dortfop motPNMt. motrko. and modoto to domoiv 
ttroto tfiM tot f|i w om con tiondio frowto to domtnd 
up to toroo tlmot cufTOfd tovtto." 

UPOa CAAift Opararwa CaaKM* Vihdaitoa.*” SyMM 
CapadCT PUnmad lod toftaaiMaL AmpKt MaiuffiMM 
Labamary. Aoapaca Radaalfn) 

3000- DwaniunaoifaotyinaatialPO'OoifTawtkwCa 
3010* D f K o tti at a tapaocy m€taaaa ta 14d% c w tn kwh. 
3013: D a n w nrti a w rapaaty mtaama ta 3300 c un — kwh 
301S: DwanrtiHi tmulp >nciaaaala>000 curmo Wtab 

Environmental evaluation 

Ofotop motood. moirict. tod modoto to docnoo- 
MMto tool oriotton notoo ond tmUtfom coo bo ttg* 
ntocootly rodwood to obtoHito tormt to onobto too 
atr Irtffk tyttocw lo hondto irowto to domond up to 
toroo Owot cioroM i titto.** 

IIWPOMM «d bariy.*** Jpoa CMS& OprtMaM Cai^ 
capt VihdatiaaJ 

3000 Oamamtaaat no nniaaaMoui laatitilioni at 130% 
capacity 

3010 PwniiMp^i nn ww%iannatai laanietMoa at 100% 
capactty 

3013 OfBaaHiaa aa wctraaoiHii taatttctwoa ai 230% 
capadiy 

301S Piatanciwaa na cnciraawfil raftncneoa m 100% 
capMty 


"fia 

•*Tf« 

•nmm 


m^atWtim 

rn^ammsm 

m^a»9U» 



t Mf aa Pa. fab. aM aPM ^ 
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Fthibit #11 


Mtioni AMWOon pQ^morm Mon<tonrg S<»r.<»-Ajf Cfrttr PtM Sufvty (Vtr 15. 2(K». r0Ca»0O4) 1 



SeCDON A: BACKGAOUNO OueSTIONS 


nfmooucnoN; 

For two our^ moot of tho ouooOono «■ raior to OOnW) day* prtor to today Tharatof*. wtwwor I tay tw 'M 
(TME PBMO). I am ralaimg to too partod l^om (START OATq tfwough (END OATS). 

I am nma gomg to am yoa a <aa R j iiio n a aftom lha co mwa mi ai tytog tial you drt during t>a lato (TiMe Pgaoo) 


A1 Ouftooffto tool (TKtEPefOOO). hoar marry houradd 



MOMFr^jeoAvs^fM iaoAys»Ma npttm 
tonriy Youmdy«uihw(MOui^Aithoun 
dtftop IM IM (TIME aenoo; aa a craMmambar ori 
a coaanaroito aicraA to Ma corracT? 





RR. m 

r 



Af Ourto9fftotoof(7MefCRyoOi;tltoiafigiMy/!^^ 


MM 

jw^MacwMwiamtMonaccmmaroitoaBfCTaff? 



Oi 

m 


A2. OtfVigthatotf (TIME PCFttOO). hoar many laga tod 

youlVaaacrtoivmambaroneommarctolaircraA? •iaoMiTt«awoo. 


1111 


A2.1 OurtogtoatoiKDMiPCRiOOkhtoanianyoftia I • ■ 

(M2)to9ayoutaarmQMadaaunoodortontonQai •itoaq/ fWii A I I I 

anaaportomaidathaUnliad gtain? ttJMMRtfLeo6MA2.iMU8TacuaaT»«wieiieQuM.TO 

LBOOMAL 

aotl: tt« unmo fTATta MAM TW to tTATta AMO 
WIMMOTOM DC, aVT OOtS MOT MCUIW W 
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A>Wbon Optfibom Montoftng S»fv^ ArC«rrif^lotS«fV^(VfAC^JU>y ^8. 2003 v 002-004 1 Pao»2 


AS. P>— leifn»»>miMAmodt>«ndt<nwtBr<lc^»»«ircn<t|m inroommf cia Wyf crw wiimbf dtftng 
m» leM rraie pcmoo)? record verbatwi m coiumn a. twn ask prompt 


PROSED AS.A f : Oitf >0(/ Sy any ctfMf awAmi 
modt/t%vmhm ofiMncrsRo o nii wiJ aiy dbrwgtfw 


«o 


• 

m 



m 


« 


PROASPT Aj^A2r PtoAMltf mvlfWAwfaPcnff 
mstoft modisfsridMrfM yotr Rpwcorwmtu M iyRPS 
0 «miN«tt*r SW Asm PERTOOp RSOV^ 

R> 0 0tUM¥A 


8 


Dtf«1Q tfw M (TMC PBQQOX 
«t« ptresm Of fw 0^ IN A1) 
«youlytit(MAi® 
MOOeuSBSCS)? 


A UAK&MOOeUSeRCS 

(NOTE. MAKEMOOejSCnES DROP 00^ 8CASN PO^EASB) WITH 
TMS VERSON) 





MOofttlAvtiUonOpOTOomMofitertnoStfv^ AifCim<rPaot8tf%^(V»f AC»>J^r ^5 20C3 . 032 054> PiQt3 


IKTROOUCTKM: 

Mng M (TME PEnoD), you may tmm iranaportad pm t 9 f ^g9 r % or eorgo. or oondudad oViar fight opartfsom 

Wfe «RMld Hia lo unMlano typaa of oparattara you iow. 


A4. OurvigihaiMtcnMimtOO). whatparcartoftha 
M Ai) M you fly at a crawmambar on flighta 
aah rauanua pataangarv^ 


AS. During aw loBtrnuEPCRiOO). what parcant of tha 
a< Ai) M you wort aa a crawmambar on Iflghta 
that camatf only ewpo or flatghl ar>d dW not cany 
rawanua panangara? 


aowoofaBgffWPinwnnmi MM 


«NomawaatoRCAAOo-. _ I— 1 — LJ 

mt rorAi nawcaar OP M AA AMO A« 

BHOfAOMtm 

A. WhattypaofSghtawaralhaaa? 

flPCOPr: 


Ae O^Ptngthalaal(TMEP8tioo).whatparoontofttta 
(HRS M At) OU you wort aa a erawmambar on fltg^ 
that camad no ravanua paiaangan or cargo, auch 
at mamtananoa fltgWa. tarry or rapoaiionng 


A7. During thaiaat (TIME PEWtOOK did you fly a oommar o al 

attrafl (READ QueSTtONS]? yes m m OK 


a 

b 

c. 

d 

a 


aaacaptain 

aaaflratoAcar 

aa a (bgtit angptaar or aaoond of 
aaaraiafplat 

bi arty othar capaedy (SPCCipy) 
1. IMiatwaatiatcapacty? 


t a ? a 

t a r a 

1 a r a 

t a r a 

1 0 T a 

Alt TwaouarrAy cAMworAuacAMawwapMO. 
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NtBonilAv1iBci><)p>rilOn>MofWlcnngS»^ AifCamf P<iotStfVfy(VyAC^Iuiy16.20W vOCC-W) ^^>4 


A71 VIMG^efVwlblo«M^tfv••cal•9onMb••t 

doacrtm 9w fKvnbtr of airpifenM eurrtitV 

cptf U id by youf iirtn»7 PImm <io not inolud« 
•irpiinM opbratad by cod»-«fttr» partnort READ 
CATEGOfUES 

MOtr m Ml ONLY NT1RI«T» « AMaUMOb 

eumiwivv UMO uHa MOT TXOM M trofuoe. 

MIOMVa^aUW POM WMl THAN out AIRUMi IN 
TMtPMOO: PMWIliWNfHMji O MOt i aiMficUTWay 

caMM «)r »•»*>• M HOT IM* •• «MI HAW* !•» M r« 

MCTMiPtNIOO) 

A6 ApproNmaiahf' now many boNramiDtalhaYa you 
flown a commafcW akcrafl during your caaaar? 


MO— TOTiti^w 

rn^irn mpmm . 

OT ^ 
g«C 


TorMHOumcumocMCSK I I I I I I 
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Niflonai A*<iDonOp»ridoo«IUk»oaQ*W»o8<»v>c»-^C<rr»fP«M Survey (Vtr AC^hA 2^ v002-004 Pa^ft 


S6CTKM 9 : SAFETY RSLATED EVENTS 


I WTHOOUCTWH: 

^ Mynart »i<i>ona if about Mfti»yr»»fd#w^nto. IB ar wB fWigttaw r MaaOcfta. pit— you 
j oKpartancad oo a c o www ci al afco«afft oo wh*eh you wra a c m wDa mb ar Tha flrat of thaa* quaaUcna w* about 
aqalpraant falNad avaata. 


ERI HowmanrDmMdunngMlMlCTMePEiaooi •eou^^xtfooblems Mil 

dd an arcraft on wNch you wara a 
erawmafnbar ^atl to an abamaia airport or 
ratum to land baoauaa of an atrcnA atMpmant 

K Vttiai ayatoma cauaad tha dNaracn or ralum 
tolarto? 

SAfcirr 


ER2 Hew many anwadurtootoaloal(TMEPEllioo)dM 
an aacraS on wtKtfi you wara a craa a na m bar 
aotoonanca a apM, flra. fumaa. or aacraft bamaga 
dua to iranaportlno Noardoua matortaia? 


A (Hoamanyoflhaaa|finER2]timaawaratha 
ipAa. toa. ftanaa or abp^ damapaMtoi tMa 
trp, fUnao or adcftS ddmipo) M fho 

cargo corTpartmarc? 


B (Mowmonyotthaaa|ainER2]1imaawara 
•pAa Ara, Aanaa or alroraB damagWWaa thia 
toa. Aanaa or aireraA damapa) to tha 
paaaanpar comportmant? 


^ many ol thaaa 6* tw p a) Omaa wara t>a aouf 0 FCOyKiNcewn>iR 6 OUATioas_-LJLJLJ 
•pda. flra. fumaa or aacraB damapa/Waa lha 
•(A Ira. lanaa or aircfal damapa) eamad 
bacauaa Via natardoua matortala In quaatton 
wara out of compi ton c a wdh rapUMona? 


ERA How many imaa dunng Via tool (TME PCmoO) did I I I I 

an aircraft on atoich you wara a c r awma m bar 
aipartanoa a cargo aMI 


ER4 How many tonaadunnp Via iaat(TlfgP€RK)0>dd an to-flgw aircraft on wtiich you wara a cr awmarrbar 

mpmmcm unoorm ia ndad m ouamanta of any of Via teaowto p dvrtcaa (REAP OUESnOMS}'? 

a. Un c o nanandad moMaman to of tha ■•aiaiora? ^ taAVATona ^ ^ t ^ 




• HA2MAT Mil 

toCMPTOtoH 


■ wCMOocoatoMmvrr MM 



• MiMdacMQiAOOiMtrMaMr MM 
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N<>or^AviiBwOptf»liOfwlylo^^ Air Cafftf Plot 8>rv<y(VtfAC>lu» IS. 2003 V 002W) Pao>6 


d. 

•. 

f. 

0 

h 

L 


Unoornintndtd fnovwrwiis of tfw •poOtrt? 
UnoonvTiflnd#d mo¥WTMriB nl Cfw ifiMdbfitcM? , 

Unoommandtd ol Oi« mm Mm? 

UnoommardM mov*rrMnl» of tM lifM? 
Uhoommandod mownanti of Om amM? — r-- r 


$ 

0 



• TMMTABT 






L 

1 

L 


1 


CM aiyy cttMT dOMOM hovo unoommondod 
mowimen u durtnp m» M (TME PCmoO)? 



1 . MNcIi doMcoo^ 


d p ccr y. 


2. roll BACK OeVICeUSICDMEIMH: 
How mony timo* dU (OCVICE USTCO 
IN EMI) iMHOrm uneommondod 
mo^monM during thd iMt (TME 
PENIOO)? 


I I I I 


CR$. How morry timet durng the tMl (TME PEMOO) 

did dn irAgM eifvaE on which you wefo e 
ormwnewMef eo^enenoo tmoice. flfo. or himet 
tial ortgineled m any of tM following MM 

(READ QUESTIONS) 

A- tht engne or n eeele? ewtitOMiOWMcaxc till 

dt^moeTotfiM 


1. (OfmepliftERSAjtImMihMWM 
•moM fire, or fwvtM in tie engmo or 
neooBe. how many iriMlvedOid tio 
tmoke. ire. or furriM in die engine or 
r t eoe ll t tiuoNe) ele cf i rtc il oowponenee 
orwmng? 


eW 0 MEW«IE/FlME 8 _ I ^ t t 


fMV AWMerr M MA f CAJMor M ohM m? rNMf mi 
MiocerrwsfM. 


B UvagMOwk? I I I I 

deMPTOEMC. 


1. (Ofthe(smERS8)BmMVierewM 
■moke, ire, or lurnM In the iighf deck, 
how meny InvokredrOid the ■moke, ire. 
or kimo* n the OgM deck inwhe) 
eieemcel components or wiring? 


meAMOowrnrsiieer CAiiworeffdMMniwTHAwrNt 
AMOtemwiiifai 
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H«boBi<AyiiiionOp<fi>ontMonao wo oStr^ AifCarr^tf Plo<8urvwy(Vtf AC-Juty »ft.2003 v.0(tt-004) 


C. tfw cargo holtf? . 


1. (Or9ttpl4nER5C](imMthMMo 
•molco. Art. or fumMlntho cargo hold, 
how rnonir »vwohf«d/Did tha fmcAa, lira, 
or Ajmaa in f>a cargo hold moiwa) 
a^actrlcai oornponarita or vwrtrtg? 


iatCMOOMOifi 


I I I I 


rawTomo. 

1 1 I I 


TMv Atfourr M OTSC 1 CAMvor M OACA raa nMA rw 
AAiouMTarciuc 


D. ttaipiay^ 


(Of tha |i in ERSOl timat thara aat 
•moka. lira, or hjmaa m 9>a giAay. how 
many invoKadrOid tha ameka. ir^ or 
fumaa in tha gaAay nvo^) alartncii 
oomponantt or ««v>q7 


^ . I I I I 

paarakfocaat. 

I I I I 


THt AMOUVr «r aOMH CAMMOr M OACA na THM n« 
AMOcarrMWtD 


E. alaaaAwra in Vw pataangar oompartmard?^ 


pawokTOffiiF. 


(Of tha (i in ERSE) Bmaa thara wat 
•moka. ftra. or fumaa abaahara n tfia 
paaaangar compartmam, how many 
twohwdOd tw tmoka. tra. or fun^ 
ahmaharam tha paaaangar 
oompartmarrt Inaoara) aiaolrical 
ccrnportanta or wiring? 


till 


TmAM0mTmm$9i CAmoTMOfmAfmnuMrm 


tw laat (TMC PCMOO), how many 
imaa <M an inflight aacrak on which you wara 
ac r annwmbarai^ananoaamoka. hraor 
kimaa dwt orlginaaad othar than at lha angina 
or naoaka.fRgN dock, cargo hold. giAay.cr 
paaaangar compartmant? 


a oaxwAtc on«ii puca . 


111 ! 


1 . Whara did tha ameka. lira or kanaa 
ongnala? SPECTY 


ERe Dtftng tha mat (TMCPCAIOO). how many Madid akMC4iuroMrriNoa«aMtfTgoi^ i I I I 

an Inflight airerall on whtth you wara a 
erw w wamoarakpananc a a pracaubonaryangma 
ahuMown? 


ER7. Ourtrg tha tact fncPCRKX}]fwwrnwiytniaa a totm. BfOiHK rAuM t I i I 

did an infIght aircrafl on wMeh you wara a 
crowmampar a> 9 arwnea a toM angna ladura? 


/■i n wn 





INTROOUCTKM: 

Ttm lalowwg qumtnom nttm to twrbui»ftc< 


Ouneg th« M (TIME Pe8»00). how many OmM «d 
on akcnft en which you wm > e i owifnw f tbof 
(REAOQOCSDOM)? 


TUI. EnoounlarMvorolufbulonootfMicowMd ttrgt 

«bn« ohonpM in •Uudt. anpood. or atttuct* 



A. (Of 9 m lAn TU1| toinro ImtMilonco 
onoounM, how mtny oocurrodOd 9«t 
Mwar* lurtMiMKo onootfilor ooour) in I lA C. 
condMont? me. • mstrument 
MET f OAOlOOfCM. CONOmONS 


B (Oftho{f lnTU1|«ov«rokifOulonoo 
•noourttn. how morrir oocurrodOid this 
•ovoro b^rtodoneo oneotonor ooeuO m cltor 

m7 



TU2 EnooirdirwMcotutuianoothoifOMiBadkniOor 

merodogroMoAoircrafirol — ffitMTMOMAiiaudTncu I M 1 


INTRODUCTION: 

T>io nox! law ouoationa ora abool w—thar fotaaod ovanto «d)Ba akboma 


Oumg tha loaf (TIME PCRtOO). how many Omoa tfd 
an aatrail on wfach you wara a crawmantwr 
(READ QUESTION)? 

ME1 LaMt ac cu fUa aaatiar rdor ma fton afhan 

craamambara naadad t wdMa aAboma ...... 


A. (Of the (9 WEI) ttmaa whan a awnM Hi Oai a 
MOad accuraea waathar i n lorma ti on whia 
atrboma. how many tnvohwd nofvU 8 airpoda 
or oontrolors?/ Old 9m Mna whan 
erawmambara ladcad aeeuraaa waathar 


auCKWEAT>«a»Ct U MTION I 1 I I 

waanaioafix 


••MOLVENOiiMA«aomoncoNmou0i I 1 M 



Mlonnotten wbla airtMma imMNa a neivU S. 
atrpoit or oo n trodar?) 
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NWonil AvtMi0BOp«tMiaii«McntailngS«<vtc»-^C«nl«rP«0lSur»»>(V«rA<>July 15, 2003 V 002-004) P«9»B 


licfc»d texunta wrnfm tntefwaen w«ii» 
Mrbcni*. h9w many irw«*Md Al^AM this 
tim» iirfMn awrwmteri tocl(«d aoeurM 
mf KN f tHof mt tteft white aUtxxrm irn^civ 
AUST) 

Fa« 10 fOOOMo A.T.C oppfevtf tor • foguon to 
■»oto — w woothor 


K <Oftho|> WE?]>niM c r — imombontofcdto 
roottvt A.T.C. opproval to ovoid mwo 
nvthor. how many Omaa ww wnorgtncy 
omhorty tnvokod ir> to—a ntua t o n aMto a 
arnargancy authorty arvokatf to Wa itoiallon? 

Dtvart to m ata m a n aHfaid bacauaa of 
waatoar 

WE4. Eiawnanoa airf^ama ctog toot raPuoad to# 

WreraR'a abOty to maadain tottuda. ipaad. ttotoMy, 
ortoractionaleonOol 

Ef>oountar vw r d ih aar or a na ci cburto oondtoon that 
raaiJiad to an aaapaad divialion of 1 5 totolB or 
9»«tor 

Vl€9 EhoountoratodihtaforamicroburMcoridKcnthai 
raauAad to a wtndahaar arvoidartoa martto^ar 


•aMOuiAm Mil 


rmjmotmrmmtAAMomit coitoaoB) 
Duwor at oaMna mM 
rHtAHouwnawtf 


arMLOKSMtATCMtoROMIL I I t I 

to 0.119 TO wca. 

•BOBMiBicvAi/rHoanvafvOKeo 1 M I 

TmAMotMTmmiACAmtoTmwArwrmj§nm 

AAtoummmz 


•owtoTTOM-TBatoTiAtotoao.-.. 1—1 I 1 


• cnw w paaw AAMaicaio. . . _ .. 1111 


•ENCOurrBtina«)toCAAiaciioautotT 1111 


lIVSULTWMNOto^tlAVOWNCi 1111 


toA4«O.SKiPTOAC1. 


HfTROOUCnON: 

Tha mod tow quaaborw ara about pa a aa n ga r r atotad awania 


Durtog too tool (TMi PCMOO). how many Bmaa <lid 
an IMMgM aiortol on wtltfi you wara a 
erawmamOar {READ OUCSTK3NS): 


C^. ExpaOiatortotogorttoiartloanaaamatoaaport 
dua to a patianpar motocai amargarwy 

CPZ Expadlatortoing or dtvart to an aAomala airport 

CP} Ottog too toM (TVie PCfbOO). how marty tlrnao dto 
a crawmambar tofva too eockpa to handto a 
paaaangar dtoturbanoa on an totogM aa aa h on 
which you wara a crawmombar 


a M TO hAsacMOOT uBNCAi ocnoercv LJ_LJ 

aouETOhAsaotofRorsTUMUMef 1 1 M 

LLU 


a CREWMEN8ER8 LEAVE COOtofT 
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NaBonal Av<»on OpfUtoftt MoMcrtng 8»\<C< AtfC^rrXf Piet Suvty (Ve«- AC Juy 15. 2003. v &02-004t P»g»IQ 



OurviQ th» Iwt (TIME PERKX>). ho«r miny Qmm dW 
Ml mitnin on you twf • tfnwtwnbf 

(REM>OUUTOO> 

ACl. ExpMtwiceabMtarK* •»M)5TWEe.„ i t I i 

AC2. PMtormMiavMMaeOonloavoidMilnimiranlbw 

fligM colMon wMi MKNhM Mrcraft tfMt WM navM 
doaer tan 600 fMl nctuMng avawa acton in 

raiponaatoaTCASaiMaofy? acvwyEACDGNB I I 1 I 

AC3. Expartanoa laM Van 500 M or aapMMon 

tomanotharMrcratwMabotiaircrataara i i i i 

aiftoma titsaiHANSoonra^AiuTiON Mil 


INTROOUCnOM: 

Tha natf raw ouaationa ara abod ground oparabon a 


Dutog tha Intf (TME perioo). hoo many bmaa did 

anairc »M <onatiitf>youv>araac / aama*rbaf 

(REAOQUCSDON)? 

061. Go off tia adga oT a nimvay or taxway wtiio 

laxino 

062 CotbaornaorfyeoMawffbagrDundMhidi? 


A. (OrtiaIfflnQE21naaf c oaaio n ioaha 
ground vaMefa. hoar many ooeunadOid iNa 
naM ootaion wth a groi^ Mhiba oeeur) 
afda yoM mrtnH waa on tha iwnp. MKon or 
inthagataaraa? 


aoooar axx RuMM^Y/TAXMnv III! 

acouJCtwrrMOwouNOVpicu: I I I I 

taLaiaatooax 

fONaMMWtaOHQATEAACA 11^1 



B (Ortiat*nOC2)naMe<«M)nt««ha aoHTAwmv MM 

ground vaMcla. how many oecMraMM thia 
naM ooBaio n wgi a grou^ vaNba occur) 
whda your atcraff waa on tia taMaray? 



C. (Ortha(iin062|naMcoAaiomtNttha aONfuno^v till 

ground vaNba. how many oecMradiOid tNa 
naMOoBaton w9) a grou^ vahida occur) 
afhda your aircraft w«a on tw njrway? 



063. Skid, afida. or hydropMna raaiffing ki a aigniAcant 
incraaaa in a l oppmg dManoa during iMidkig 


064 Expartanca a raja ct ad tafcaoff 


t REJECTED TAKHVFa 
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GES. GoofflhaadetaliniranyaititaMilngoaar 


OE6 Oo off lh« and of runwajf . 


• 00 IDOI V ffUMWMV ^ 


• 00 OFF eM> or nuNmv - 


<3E7 i ftt dvooowtty onttf an octtwo twwy^ 


FBOBIACTf^nUMMAy . 


069 BoonttMoffraflwtdoflnoffwiircnflocoupiidar 
WM ORMMng tho Mmo rvwoy 


069 Land Fitiio WMChor aireroi oocupM or wM 
croMtno Ihi Mmo ruTMuoy 


• TMCeOFF AOU MTTM occurco RUNMMV , 


• U#0 ON OCCUFO MUHM^Y 


0610. NMfly osportenco o ground ooMon «ffh onodwr 
NrcftflNNIt both oirortANOro on th« ground 

A. (Ortho|9iftGEiqn«woolNiomw» 

•TKMNr oirenff. how many ooeurrodiOid mo 
nooreeiliion Nth onothoforgiA occur) nMo 
yoi# oireraft wm on tho ramp, apron or in Iho 


a NEAR CNOUNO OOUiSON 


aONNANPMPNONrGATt AhCA 


va. OK# TO AMI. 


M AMOuvr av otioA CAawor M 0N8A ran mur n« 
AMOuararotrai 


6 (Of tho jt in GElO) naor eciaiona 

another airefoft. how many oceurrodtM thia 
near ceffmon wth another preraff occur) whta 
yoir aircraft waa on Via tamwiy? 

C (O) the [• in 0610) near eoffaiona wl 
another aircr^ how many oocurradM me 
near eoNvion vyfth another iRriff oeeur) FffHi 
yoir aircraft waa on Oia rurvaey? 


TNf AMouvrar oetwi AMD oetaa ccaiBaKco cANMor 
Off OMfiA IBM mur me AMOwirr ar oera 


TWAwotarrwdiiM. otm.Juioogf0ccomaiu 

CAMMor at OMCArtM mAir TM AMOUNfar oera 


INmOOUCDON: 

The nait few quaauona are about atrcrafl handling ralatad « 


Otfing the lait (TBIE PSbOO). how many »naa tftd 
an avcrafl on which you ware a aemmornbar 
(READ QUESTION)? 

AH1. Uaaaomaofftafaiervafuataadaine d byiha 

FAIU 


AH2 Accept an AT C claaranca met tha aircraft 
could not compiy erffh bacauaa of ts 


AK3 Uaa tight of another aircraA from wtaoh the 

atcraw waa trying Id martain visual 


A (Of me ^nAHS^timaa an aircraft teat 
alghl of another aircraft, how marry 
occurradOid loaing sight of anoffiar 
aircraft occur) m maiginal visual oondtiona 
ofSmasaoriaM? 


SAfiCerr CUAAANCK NOTOOUFlVWrTN . 


a LOaE aOHT OF AaCRAFT - 


a WMAMOMALViaUALOONOrTONa- 


VAWTOAMA 


TM AMOUMTaVANSA CANMOrae ORCAnW INAW TMC 
AJiOurrwAia 
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AH4 mt^tflwiOylvtd Without dMranc* Man airport 
»•) on oc^ control toiAfor 

AHS I no chortontfy bogri ttfcooff id wOhout A.T.C. 
cioorsnco ot on oi^wt wNh on odho control 
lowor 

AHS inortvortonSy doutito from on o wy od foUing or 

A.T.C. vodor lor ono mnulo or mono 

AH7, Exportonooololslrtkoontonrting . 

ANS. 

AH9 EapartaneaohOfdlanelinfl 

AHIO. Tolto off wth on out-o t mot eontorol yo»rty^M> 

AH11. Toko^ovwwogM 

AH12. Commonco toko^ fol wth on imprepor oircfoft 
co n S o u rooon 

AHn EadortonotonungmoloRturtolorony 

roooon 

AHM. & 9 ortonoooviidotolwomingoriCidishoicor 
ocOMlion 

AH1S NoortyooOiOotNrthtorrolnorosrour^obolrucdon 
whitoitofno? 


PfTCKVKMCJt: MCLUOM BUILCNNOS 

A. (Of tht|t In AH1S|rwooMono won terrain 
oroproundobotrucbon, howmony woto/VMm 
two noof coition with terrain or o growid 
oboirucOon^ProuQhl to your oltenUon by 
A.T.C.7 

B <Of lho|0 In AHt5|noof o o»ito n »wOh terrain 
or 0 gro^ obotruction. hour mony woto^Mm 
ite rteor coBoion wllh terran or 0 ground 
otwOucSon) detected ttrough dboci avhdng of 
tw graund or obterucUon? 

C. In AHISj note coBoionoWh terrain 

orogroteNSobotniction, howrnnony worteVWoo 
two note ooBoion ortih terrain or o ground 
obobucdonydotected tirough octNoOon of 
aP W8. or E.O P W8.? 


• LAND W«0 CLEMANC F l^U 

• T«l<EO>ri(QU.WtOafA(UMCC I I I I 

•otvwnoM _ i 

• TMISIMCSOHUINGM I I I I 

• TMLSnmSOMTMEOFT I I I I 

• HMIOUINOMa* I I I I 

• IMS-on our-aMJ«itTcmBio»aiiAvnvl_LJ I 

•nmen ovERweoHT till 

IWmiMWWICBIWUiMICW — t J.J.J 

• HRUaUH.AT11lim 1 I I I 

■ >T»U WM Md S n e n XMPI «CTW*I10M I I I I 

•>itMcau*o«»ia<*oi«c L 1 .1 J 

viiM»To«ai. 

• AICaiOuaMTtOIOUMATIIHnOM I — LJ 

mtAMOUHTMAMTlA CAimOrMaiKAIW miuinc 
AMOuriMAmi 

itiguctiDiwweueMcwtctiwwiia -I— 1—1 

INC AmOWrlM AMH AMD Am» COMMOO CAWlOr 
ae 0«A m INAA na AMOMIT W ANIA 


•uTTccTCDivctouaHapiimcaPMs -1 1 I 

mCAaOUHIMAmAA AwmAAOAMKCOiaMO 
CAMKOrW ONCATIA IMAA IW AHOtWI AT OtW 
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1 (HownwryorihM«(0inAH15c)nMr 
cc»tio>n wt f X Ww tMt tmt cOMon) 

(Mtotod ttrou^ ■cttvt&onof 
EOPWS? 


INTROOUCnON: 

The nm <tw quMUons am about AmiMk tfaviallofw 


Hew nmny ttiwa durtng tha laal (1WE PCmoO) did 
an mrtnft on wWeb you ware a eraw m a w bar 
(PEAOOUCSnONS)'? 

A01 inadmrtangydawaiiairomana aa yadataudaby aALTmmocvwTOm ^ I I I I 

mora than 900 «aac? 

K (Ot0wI«*iAOl|daviationabemanaaagnad awwotfiTOlCM I I I I 

aMuda. how many wa r aAlWaa 9aa davtabon 
from an aaagnad aftituda) n raaponaa to a 
TCAS RaaoHjtton Adnaory? 

A02. OaaoandbakwMMfTvn Sola AMuda wnany^ 

warn Mdloaowng A. T.C. radar vaciom^.....r»M a MnrauiMB«>ATCRMMR ACTORS « i 1 i 


MTROOUenONS: 

Tha najd law quaabona ara about intoracbona wMi air traffic control 


ATI. During tha laat (TMt PfMOO}. how marry umaa auHiaUTocowNUMCATiwmiArc . 1 — L-LJ 

waa an aircraft on wtacb you wara a erawmambar da.aiQaTOATa. 

urobia to oommunicota wOh A.T.C m a tima'Crecoi 
ahuabon bacauaa of fraquarKy cong aati an? 

Thaaa problama may haua oocurrad on tha greund, 
or whia aaboma in tha tarminai area, or whMa an 
routo. rm going to aak you about aach 

A <OftoaaaCliATl)timaaycuwaraunabiato 
oenwnunicala wtth AT.C. in a ti ma- er tb ea l 
aduatton bacauaa of fraquaney oor^gaatton. 
how many ooeurradM tha tona you wara 
unabia to nonranunicnia wfrh A T.C In a tone 
cribcai aituabon bacauaa of fraquancy 
oo r >gaabon occur) wtida on tha ground? r.n.-r 
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B (OHhaaa(t>wATi1Jb m » a youwafauiiabialo 
oommunicaia wtfi A.T.C in a ttmn cf^cal 
ttumor\ bacauaa ct Iraquaftcy conpaanon. 
how nnany oecurrodtOid tha lina you wara 
unabla 10 oemmuraeaia «fth A T.C in a 
ortcai otuaOon bacauaa of fraquancy 
coftflaa tt on ooouf) wNk aktema ki ttia 
tanailnal afoa? . . 


aMNUMVCAtC. .. . .1— l—LJ 

amiEs 



C. (Ofthaaa|iinATll]tlmaayouwaraunablaio 
oonvnuracata Mh A.T.C. n a tima orticai 
ailualon bacauaa of kaouancy oonoaation, 
how many occurrad/Did tha *m you «ara 
unabia to oonwnunicato wth A T.C In a ima 
crftcal atuatfon bacauaa of firaquancy 
c c nQa aB oo oeoa) wMto an roula? 


awHicetnouTc 1111 



AT2. Hoar many tlmaa dudng 0>a tan (TM8 P0bOO) did 

an airaii on aituch you waft a crawrmambar fiy at 

an undaairably high aiPtuOa or ampaad on i i t i 

appreaeft dua to an AT.C. d at r an o a a>aoMAi.TmjOdOAmywn ... I I > I 

non TO amawncM; tmh aaetueci at/r auv 
aoT M uantD TO 'mAH omar Apanoaoca. 
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N«Der«»yniionOff«>itcn»Mon«o«in9 8«tyte»-AifC«mifPilol8KVY(V«>AC->^1S.aO(tt.»002-00«) Pig»1 
SECTION C: wcuwe AI>mOACH CHANOES 


S4TRO0UCT1ON: 

My n«d quMtMm ara about c laaaao c a cha n gaa raootvad on approach wNwi 10 nalaa of tho naaatoy Owaohoai 
that UiaflisMcnw did noanquost 


Oumg do laat rnUE PERiOO). how many bmaa did 
an aircrMI on which you wot* a CfWMnambar 

•acoiva an u waq u aa l ad cl aat a n n a change lo 
natway aaaoninaol, aUuda laaincbona or 
anpaad wNun to niSaa of tha ninaioy dinahold? 

ROUTE A— OfR-Y ONE CHANOE 

K WaadMiauoquaotadcaaaianoachanga 
daeSnad? 


S Old «aa unto a u aNaOclaarancataianoaiaaul 
bl(l«AOOUCSTIONS)T 

1. An taiatebdead approach 

Z A goaround or naoaad approach 

}. An aatxima oonnet 
4 A 

t landr^nlheU-oMinlllaAMndaar 

0 Landngcnannnonamsy 
7. Landing long or laM 

a 

S. AoonSctonlhagreundwtdanefhar 
anoraA or graund vaHcla?. 

to Arty olhariaidaalr a tfaavaofadlar tha 
daaranca changaT 

a WhN arania ooounad? 

sPEcmr. 


a UMaOUUTEO CI£AIUMCC CHTWOEa 


-LL 


V Ot. OK M Mf . MV TO MCnQM 0. 
W *1. CONTWUC WrTM noun A. 
» a OK liOM. MT TO Mun 0. 


vn 

CIKrTOfBCTIONO) 

QQO 





#iaK1OMeTKM0» 

W* 



MO a 

_ss 



•K#TO)C2. 
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ROUTE B— TWO OR MORE CHANGES 

A 0(ih«(»iwici)unr«qu««l«d«M ww»CT iumnunttooEMUNCZCHNiaes I I I I 

IK»» tiMtly, » »iy. — 

awToacnONo 

» OMLT OIC CHWIOE KEHMMA 00 TO nai/n A lets 

I THEWNIBa»OFMWEOC«STWa*UMNegCHAIIIBWIWS( M UWit W «f;SOrMi«MMiW^ 

lAMgQIiESrEDCtEARANaiCHAIWesmiirWOtea gCK M n WASroSEIWUMMRWieUORREWER. 



S^OFY:. 
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Nldon<IAv<ti>owOptc«Bof<tMon< IO fin^ AgCirnf PtolStfvgy (V<r AC-J^ IS. 2005 v 002«C04) P»o»3 


WTROOUCnOM: 

(Mjr nad quMtlom an about iMt •ocaptod ctMfonca cMnpa M an iwn boan taking abouUWy naat t $ma tr<nt 
an about 9m moot rocont clwmaca chango that tha fRokl ama a cc apta d .) 


IC2. At aNch aiport did 9m avant oeaa? ^ *macmi 


K PlaaaaMmakiatocabonidaoaAarfer AAboarioCATtONO 

(MI^OkT). 

IC3. ASK ONLY a TWO OK MORE WJCXIJ fVPOinCD 
M A3, a ONLY ONE MOOCU SKIP TO IC4. 

VMatf) moda aroraft wara you tying wkan ttW mM a a mrai . 

want oeeuTfad tha (U8T MOOCU tN iOA)7 OOOC 
MODEL FROMAM 


IC4 Wart you a c f wanannbar on an FMS or PMC 
aqugi^ aircraft at tha bma of Ihia avarc? 


A. WkadiaFMSarFMClhalwatbainguaad 
capabM of ttortng muftipia routaa? 


B. AmaanadgationaratootTirnuni ea ttcn 

Iraquancy chargaa in aw arcrat mada through 
t)aFM5.orFMC? 


tC5. bi ra tp o na a to 9m e t aar a nca tfanga. Md 9m 

Rghte j aafabrogfamofattamptlofaprogramaa 

Fli&orFMC 


vn 


1 

a 


imtmaiMnL 

* 



t 



















99 


» 


.»a 




a 



* 

OK 

CMRTOIOQ. 

a 


106. Whan pragrarrvning changaa wara mada or 

aeaiTaiad. (RCao qucst>ons)7 ym wq ar ok 


A Did aa inputa load proparfy i 

B Wm E poMftla 10 oompMa aw progranvnino 

aKhwai'aiafctitima t 

C. WiraMioftwprogt a mnangrMuta c r o aa 

chachad by olhar craamantwraT n t 

0. Wara 9mn c&m/ p ro g iwTwrwig d ft ft eu k ai i 


tmx 


ora 

ora 


0 


FTDC 7 . 




1. Ptaaaa daaedba awaa dfteutbaa 


SaCCMY. 
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A. MMchartgttfTvo^vvacftangtlromin 
I ni mro rt ■ppfoatfi to • vwuol approach? 


ICf [MV«chanoain¥ei^«achan9oftO0iavtiual 
approach to an inotnnont approach? 


IC10. WaathaararrtproprammadtoranaukMOMpiad 
approach at fia tana of tha davanea ehanga? 


IC1I. Old tha daafanca changa charya tha airgdra 
funaay aoaipn f f i aht? 


K Old lha funa ra y raawig nm an t Inyohra a 
chanpa from on# r^jnaar to anoahar parald 
f u fiaay 


IC12. Did Ma ctaaranca c ha nga changa tha araall'a 
aMuPt aaaignmant? 


ICIS^ Did tMa ctaaranca changa changa tha aircrafl a 
an paad aiai gnm a nl ? 
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NaPonal Avwtton Op»f»tton> Mo>Mtori>>g Sfvic*~>Ajf Canwf Piot$urvy (Vw AC Jt/y 18. 2001 v 002 -004:1 PaQ»8 



INTROOUCnON; 

Onem wgmn. bifaf w oonOnua. I imiiI lo ranwid you that Viom quMbona ara atfl about tha moat raoani 
kmmquaaM dMranoa clianga a«hin 10 miaa of tha runatay thrabboM 


(C14 


IC15. 


bi raaponaa to tfaa ctoarmca cfian^a. <M tha 
dohloraw (RCAO OUCSYIONS)? 

K CbangaanavtototontotodOaquancy 

1 . ConttonchatoarMyorthanawnoMiM. . 

B Changa tha A.T C cowm u nt ca ttoo baquaftcy 

C. Ravtaa tha a pproa ch brtain o 

0 Chahpatha aaptanaoprOQurattoo 

E. Dao o hhact aay of tha a i aoma n J ceraroi 
a y K a ma ? . 

VMn tha fhoht craw otvao a raaacn for tha 
etoaroTtoa changa? 

A. WaaohaoftharaaaofuQtrahfftgAO 

ouesDONS)? 



1. waka babutonca avodanoa 

Z MaiPtan w q traffc flow and a ap ara i o n 

Z Prmaar^ a ruram ay farorabta to your 
9^ - 

4 A€hargainae^rurMraya._^« 

a. Wraaviar or anno racio^a ~ 

B. Wiiiitwnttortiy 

7. A-T.C aqupmam probiama 

0. waaanyootariaaaonotoahforvta 
ctaaranca changa 

a What laoaorta wara ghan? 



WTOlCtO 


SPECIFY 
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IC16 M rMpoodbtg to th* ctofinct chang* (fVAO 

oucsnoNS)? ym no m 05 . 

A. raducalhtquiftyofeocAptccordnaltan tori 

a r«duo» •autoMnai « • 7 • 

C. CompcofniM frame watoh — 1 « t • 

0 . Wraerftoy cernprawtotdtoanyotof wy. < v ^ ! 


•toPIDMCnOND 

1. HowwMMtoty oompeomtoatf? 


tPEOFV 
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N«Bofti<AvlWionOp»ri6on»Moftiowr^S<r><c<^^CyrttfPlocSury»yf\^AC^ Pag«1$ 


SiCPOWC:JIMOATQUMTIOW8 

Ifl lt» naott Mcbon. I wfl tw nkng you aom* quMUon« abou yourHylnQ «J 9 «lane» and trmig M 8 
fViMM 10 iVTTmil oparattona witf insfirurmra ■pproachaa Aa <i»a go loraur d . p iaaia ■mA you aniwart id 
thoaaewnga»aiyou p af>onaDyai»artaftcad- 


J01. If V>a airtfaA you lew (motO dunno tfw taat SO 

Osya aq^rippoO wth 0 P W.8? 

GPWS ■ ground promHy wmi g ayvlam 

K H i aqulppad wtt a tifrair^ Od p to y . tuch 

at you ftndfiananhanoadO P W8. or 
Tanari Avoidanoa Warning Syatam. aiao 
knowm aa TAWS (laim)? 

0 Ooaa your airtrta raquira tha (arrain 

tfaplay to Oa aaiactad during tafcaoff al 
•paoAc aaporti? 

C Doaa your aMna raquira tha tarrain 
dNpiay >0 Pa t ala rt f d during aaacant 
andlandkig? 

0 ForarnaatMtarrairKMptayianol 
ragufad. doyouuauaiyuaalduftng 
talcooP? 


E. For Omaa that tarrain dkaplay ia not 
raguffad. do you uauaSy uaa I during 
daaoanc and landing? 

F. Haa !ha tarrain diapiay anparian ca d a 
map aMI on any aaeraA on which you 
wara a craw mambar? 


..OOPTOJOlh-. 


-CIMPTOJOO- 


.tBMPTOJOQ- 


-laNPTOJBQ. 

_0NPTOJBO. 


NOORNMI iBMPTOAil. 

VESORSOaETMES 

RF ffNPfOM. 

OK IMF TOM- 

NO OR 


VESORaOMCTWeS- 
HF 




NO.NaTUaiMU.V^ 

YEB.U8U«lLy_^ 

RF 

OK 


NONOTUauMir.. 
yarn LiWiitiv _ _ 

RF._- 

OK 


JD 2 . During the laal 00 daya. how many omaa did ataiES I I I I 

an aircraft on aMch you wara a crawmarrtwr 
aMdanoa a ground prodmty warning? 


a AaaaKWTOJOx 


K Waa (lha warning/ d>a moat raeani of 

t h aaa wammga) uatd? 


During thia (moat raoam) warning, dd 
you aaa tha a pproaching larrarn on ffia 
tarran diapiay baiora you hoard tha aural 
Twmmg? 


NO 

(MPTOM 

a 

m 

(■VTOJoa 



OK WTOM • 


NO (819 TOM 0 





RF 



OK 

MFTOM 

a 


0 . 




AvMon OpMm WOrMon^ S»rvto»-Air C«rmr Sur^ iVer ACN>j>y 15. 2003. v 002-004) 


Oumg lh« iMt 60 <!■)«. howr miny OmM tfd 
<n atrcna on wtucft you m a mfnwrtttt 

• MMrm^ S«l» AnuO* Warning Al«rt 
•Ho know*) •• an MSAW (ornHOM) or an 
■ Mu d * •ifwn m c«fl from aw A t C 


<Duiif>g Vim tnott r»cfi> of » ■■■ •»•■ <■ .) VMht did yoo aroMI do w ro a po na i to th« 


B (OwWioB)l»fno«lr*c«nlA.T.C. wdrnirio 

•MOfltXMPwimAtmanonhancad JP 
OP W S or TAW 8 (iMra) nctiAad? 

OPMB • GAOUND PAOXMmr WDkANING SYSTEM 
TMAB • TEfOUiN AVOlQAMCC WMWINO SYSTEM 


\. OWyouraircrafliliortoaMVi 
yowd pfowMy warni n g from Wt 


JtH How mwiy Bmao M tha tail 60 d«ira. did an 

akcfvft on aiacA you wara a orawmambar 6/ a 
noo’practaion •(iproach‘7 


A (Wlaa 6ita no^practaion approach 6oam otiues 
I n LM.C? / Hew many of lhaaa non- 
pracMon approacfiaa wara Sown in 
IMCT) 

aiC • INSTRUMENT METEOftOtOQiCAL CONOTRONS 


JOS How many tlTiaa In 6ia laat 60 daya dk> an 
aireiaA on wttieh you wara a crMwnambar 6y 
«iun-alaM»d norvpraoalon approach aatara 
tha aircraft waa not in landing oordtouratton. on 
•irapaad. or on gida-aiopa by 1 .000 fMl I M C 
vSOOfaalVMC^ 

MIC - UCTEOAOIOOICAL CONOTTIONS 
WC • VISUAL METEOMOtOQtCM. CONOfTX)NS 


A (During lha moat racaMun-ataMbad non pradaionapproacfUVlftiallacCoraoontrbulad 10 

6ia mabary to oonduel a atabftsad approach'’ 
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Ni<>onrtA<iiCg<Op«f«awwitenapnr^ 8 »fWCT-^C^fWPic< 8 iMyty(tfyAC^IUlyU^ P 1017 


JD6 During ths iMt 60 did an aircrafi on 

wtHQh you aara a cfVMfnarnbar ta 
tfMwa batwaan ^rlng a oonatam angla 
approach or aM^^own n orv^raciiion 
approach? 

A. WNeh dd you chooaa moil oAan. tha 
conMant angla approa^ orlha Map* 
OMm norvpracilion approacli? 


-IMP TO. 


58 : 


I 

f 


OONatANTAMOa 


CHoat aorm y>« sAUi . 




PK 



<07. Oumg tfia last 00 daya. how many bmaadU 

an aireraft on aPieh you wara a erawmabar ffy 
a norvpracMon approach to a runway whan 
gtrti liopi intomahnn waa Wfadabti to you? 


aracs 


1111 


a ana aioa TO joa 


A. Owing (MPIha meal raoant) non* 
pradaen approach. dU you uaa tha 
gm H op U ntermabo n ? 



JD6 (Htha aircraft you iy<Aro any of Ota aacrall no IWtOM a 

youfly)LHAV/VNAV<t«naiAf.nau)copabla? 2 * ' • - • - } 

CK SStoah a 

IKAV • LATERAL MAVTOATfON 
VNAV • VUnCAL NAV10ATTON 


A. Ooaa your grtna auar raqwra ploii to 
uaa IMAV r VNAV (l4ianV<nav) to fty 
cenatani angla approachaa? 

1. In tia laal 60 daya. how many 
Omaa dd an aircraft on which you 
wara a cr aa m a m baf uaa iNAvr 
(L-na«/V-rtov) to fty conalani 
angla approachaa? 

B. During tfia laal 00 daya. how many Umaa 
dto an aacraft on wKch you wara a 
oawmambar not iy an LNAV / VHAV <U 
na^4ia<r) a pproach whan that opbon 

waa a ■aftahii? 


NO 

(WTOiOit 

0 








u 

LLJ 


u 

LU 





anno.araaTOJoa 



Ptaaaa aiplaln adiy tho LNAV r VNAV (L-nawiV-nav) approa ch waanl ftown <durtng 9ta 
moalfacant area that ft waa auaftabla) 
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^4itena< Av<i»ar Op<>«tion< MontoiviQ S>ryte> -ArCifrtfPiio<Su<yty(VrAC<JXy 1S.2003 V.0CKHH) 


JO0 Owngth«MMt90dty«. wmanamAon 
wMch you wm9 • cra w m f n fcar oquippod to 
ntMi Ropuirod NovtQMon PoffOfff W K H 
Moodofdo. coMod R.N,P7 

A. OoM your artM dtooM to UM RXP? 


B How mony Imoo in tho M 60 days dd 

•n oo ao ll on wttich you wort • 
cj own w n n bof fty on RN P o pprooeti? 

C. DtftnoffwlMlOOdayA. howmonyomot 
dU any oireriR on wtitcti you woro • 
c r owmor n bf net 6y m R H P dpprootfi 
wfton dtdt option ww o^ooblo? 


y«6 

RF 

tMPTOJOMI 

— 1 
7 




MO 


6 











LX_ 

u 

•Tasa. 1 1 i 1 


paiio.0K#T0J0«a 


I. PtoMooxpiamwtry thoRRP. opproach woo not flomm <mod rooani timo that I woo 
avattabta) 


JDIO IFJD4>0.6RJPTO<JD11. Ourme lha taal 60 
day*, how many timaa did an airciiift on wMcfi 
you wara a crawmambar 6y a norvpradaion 
approach into an akpoit wttioul 0 U.E.7 

OME • OISTAMCe UCASURmO EOU^HEMT 


j_u 


V 2X110. aiupfojoit 


A During (Wa avanMha moat lacani of 
tttaaa awanta), wodd DM E hara 
improaad your Miiiry to land aaMy? 



JDII. Ourbtg tha laat 60 day*, hew many timaa dd 
an aircraft on wMeh you wara a craafmarrbar 
fty an inaatonanl approach into an airport 
whara pada atopa or ot>af ground baaad 
^wrtlcai angla giddanca nfonnaOon waa 
unovaiabia? 

A Ounng (thia approacMha moat faoani of 
Piaaa a pproach), waa OME uaad to 
eafcuiaia tha rata of daaeant tor tondtog? 


• TMES 


1 wmto.aitwfOAtx I 



0 




aa 


7 

OK 


6 
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tt«l)on«tAvi«tienOp«ritioiwMon<oi)(igS«nitc»--AyC«ni«f PM 8<ri»»y(V«fACJl<»1S. 2003 y 002-004) P»g»19 


JD12. During th« tMt 00 days, how many ftTM tfd 
an oircrfA Oh which you war* • erswmonibor 
lond on • (unway wthout VASI (v«a) or PAPi 

VASI * ^RTICAL APPROACH SLOPE SOCATOR 
PAPI • PRCCISKPI APPROACH PATH INDICATOR 


I I I I 


PKROcilQPTOAOIS 


K During iho moct rooonc of Rmm ovortt) 
wotAS VASI (vMi) or PAPI (papi) h*« 
knprouod Vw aircfsrc abSSy to land 

taitlyT 



I would now Ilia to ask you soma Quastions about your aMna's wnttan standard operaOng prooaduraa 
or S O Pa. 


JD1S. Doyourartna*swf«ton$.OI*tinduda 
Cordrolad Flight irAo Tarrain pravsntton. 
tomatonaa caiad C>FIT (C4S)T 


JD14. Do your airtnt's wrisan S O P» tafe about how 
to avoid drcumataneaa toai ooiAd load to an in> 
iSgNiotaofcofDofT 


JDl S. Do yoir artna's wraion $ O P a taSc about how 
to paribnn raccvary from unusual attliMfaa 
and dtoMrtura born controSad llighr? 


JDie. Do your airSna'twriBanS.O.Pstak about how 
to awad approach and kanring acddanls? 


J017. Do youraranaiiwrtoan80.PS tab about how 
to % norvpraciston approachasT 


JD18 Do your atrtnais wrtton 80 Ps raquira fha uaa 
of oonstam angia norvpraciaion approachaa 
whan that opbon la avaiabia? 


Do your airtina’a wrICton 8.0^Pa talk totout 
how to raapoad to E.OP.W^ wa ml nga? 
sopM • eiauvcccD ground proximity 
MUtMNQSYSTQyi 



J019 
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N»Oq>^ Av>tt>QwOp»f»Ooo»Mc n i l OfWoS»*VK»-A<rCim>fPlcl Survey (VyAC^ IS. 2003 V 002-004) P»g>20 


N9w I wwAd Mt to Mk •om* quMiiont about yotf racufrant traMng By rao^rant traMng I ma«n tratotoQ 
oonductod panodcaffy that w datiQnad to martato your dtito and knoatodpa 
CtARiPICA'nOH Thadoaa not toetodatranatoonorinttal traMng Racurrant training can toduda ground 
aohool. tanuiotor Mnng maiont, and any traMng condudad in tia a t r aft . 1 am going to raad a lal of 
iaau e a - For aacti aaua. ptaaaa MP cat a i that toaic or aaua «aa oowarad dutng your laat racwrant 
training 


JD90 In arfM mofitti and yaar <M you raealva 
your moat racant racurrant training? 


j_ij 


I I I I I 


MONTH- 
TlNt 


Xi21 CNd your moat racant racurranl training talk 
alKM basic aIrmanaMp? 


A. Did your moat racant racurrant traMng 
talk about normal approach 
procaduraa? 

B. DW your moat racant racurranl tr aMng 
talk about approach briaArtga? 


C. Did your moat racant racurrant training 
talk about crtlarfa for imoaong go- 
around and mlaaad approachat? 

0. Old your moat racanl racurrant traMng 
talk about go-around and miatad 
approach txacwbon? 

E. Did your moat racant racurranl traMng 
talk about amargancy or atmormal 
condtoona procaduraa? 



Non I would Boa to aak you aoma Quaabom ooneammg trair u n Q you may harra raearvad addra t ain g 
con tr olad fight inio tarraln. or C-FIT (C^fil). and othar laauaa 


J022. Hava you racalvad C-fIT (C-fK) pravanCion 
training from your alrftna? 


A. In arhat montti and yaar did you racahm 
your moat racant C^FTT fC4lt) 
pravanOon tr a i n in g? 

B. Did your moat racant C-FIT (C-ftt] 
pravantton training talk about ndnlmum 
obatructlon claaranca atti tu daa or 
MOCA imo ca)? 

C. DM your moat racant C-FIT (C-IR) 
pravantlon training talk about mMmuffl 
anrouta alWudta or HEJt? 


NO 

BNByOJDiB 

0 

m 


* 



a 

»— 1 1 1 


MM 


aa 

» 

CM 

a 

yn 

a 





OK 

a 









Wiftenil Aviation Op<fiton« Iblofi^Drtng -AirCfrttrPioiSmviyC tfif^ Jll^ f16.20Q3 y002«0(H) PiQ>2l 


0. Old ytHtr moat rocant &>flT (C4tt) W ■■ ■■0 

pravantton traMng tait abo^ grid iS, 1 

MOlUa <mo raa)7 S a 

MOAA • HMMUM O^CRATINO RADAR ALimiDE 


E. Old jroMT Jiioat racam C-flT lC4lt| J|^ • 

pftvantton traininQ ttfc abwA O.F.W.S 2* J 

or e.ap.w J? S ; 


ORMB • GROUND PROXI«TY VMRMNO SYSTEM 

EO^S • ENHANCED GROUND PROXMTTY MARMNG SYSTCM 

F. Old your moat racant C-flT IP ■ — — — — --.S 

pra va oBoti tramNiQ UMi abom aacapa w ._i ... .... — -...-i 

maaauvara In raaponaa to O^.W.8 or fw' a 

e.aP.W4waminga7 

ORMS > GROUND PROMMTTY VMWONO SYSTEM 

EGPWS • ENHANCED GROUND PROXMTTY VaUMNO SYSTEM 


0. Old your moot raoarrt C<FIT (C^R) 

pravanboo training tak about drtR down 
procaduraa aflar angina failura? 

H. Old your moot racarrt C-FIT (CRil 
pravanbon training talk about 
maintaining aRuabonal awaranaaa? 

L Old your moat racanlC<frT(C<Rll 
prrmnbon training tak about oooRpil 
raaoufca managamont. or C.fUI as K 
ralaCaa to C-FTT 4C4R) racovory? 



NOTE CRM CAN ALSO ■ CREW RESOURCE MANAGEMENT 


J. How would you rata tha quaky Of tba 
moat rocant C^IT (C-fR) pravanbon 
training you racalvad from your aIrRna? 
Would you aay R waa (READ 
CATEOORkS)? 


Cicrarrt < 

Good , , — I » 

m. I 

Poor. I -4 

VOryPoor - - - • 


JOSS Old you rocolva training apocMcaiy in 
upaot racovary from your airlna? 

NO ... 

-tSMPTOkSb 

« 



r 



.kRkiojDSb 

1 

A. In wbal month and yoar did you racalva 



1 1 1 

your moat meant training In upatc 


1 J 

Hj 

mcovary? 




8. Wta tMa training racalvad ki a 

aknulator. In a ground ocbool. or both? 



t 


t 



* 


Rf. 
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C. How wookl you rate 9 m qiMWy ol ttM 
upool roeovory troinino you roco^Mtf 7 
Would you My M wm (READ 
CATCOOmCS)? 



J024. OoM your oirllno provWo trolnlA 0 Ml 

CocApR or Crow Rooourco Manoptmont 
•omottmoo coJIod CAJVI7 


NO DMOtOmi 0 



A. Hovo you rooolvod tliio C AM tvoMilno? 

B. CM iMo C AJM. trolnino ctionpo bow you 
0100090 9io fMolil Oocii? 

C. Oo you hovo o u 9 Qoo t k»no for Now tho 
CAM troMng mtgNt bo Mnprovodt 

0 . Whot ouooioOo B O do you hovo? 



J02S. Dooo your oMIfw hovo a no4ouk mtaood 
oppro^ or powroutid poHqr7 

CLAfoPlCATlON: No f»u« moono that Ow 
oMbM dooo not apply diociplinory action or 
crtUcNo pMoti wbo oxorcNo ttwlr auttiortty 
to oaorcioo a mtoood approacb or 90 
around. 


NO 0 

tS8 {BNPIOJOiN I 

W 0MF1OJaN 7 

OR 0NrioiDaN • 


K Would you favor tha teiadtiibon of ouch 
a podey. oppooo 9. or nokhor favor nor 
oppooa 97 


JD26. Durinp tfia toat M daya dhf you porform a 
mteaod approoch or 90 around? 


A CM you racolvo any faodbacA from your 
aMMio ropardInQ thia miaaad appro^ 
or 90 around? 

Bw Waa ttiai f ao d ba c A pookfoa, nogattva. or 
both poaWvo arNf no p advo? 


FAVCP 



OK. 


NO iMPtOJOW). 

m 

RF -OOP 70X97). 

OK Mario JCB7). 

NO _^(9apioAnT). 



RF MKT 10 ACT). 

OK MWPIOJOIT). 

pomwt 

icMTive 

■OtH FOanNE AM) ICQAT1VE — 


OK 
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J027. Dom your abflno porttdpMu In thu nMy 
roporttng pro g f im caBod A*SAP (A^ap) 
alao kfwum aa ttia Awiatkoo SaMy Actton 
ProQram? 

A. Hava you boon brtalad on thta A-8AP 
(A-«ap) program? 


8. Wara you told about ttia ganaral 

purpoaa ol lha A*8AP (A«ap) program? 


C. Wara you told how to submA an A*8AP 
(A-aap) raport? 


0. If lha oituallon ailaaa ki ttia futura. 
would you aubrnM an A*SAP fA'Oapf 
raport? 

1. Why not? 




E. Oo you btIMv* ItiM Dm MnfltentUWy a( 

MO....... 


0 

A-8AP (Aaap) data la adaquataty 

m 

tOMPTDJOCff). 

1 

protactad? 

OK 

#Kipfomr)„ 

• 


ClAmCATK)M: ConfMantlality rafara to 
both cha rapor t ar and to tfw uaa of ttw 
data. 
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Nibon<< Ayiatkw Optf»bo>^ Moo<ixi^ Ar Camer RicK StfVy (V> AC-July IS. 2008 y002-00<) P»p»24 


A. Ar« you awwo ci any pooKtv* cKanpaa 
that hav« raoutod fr^ this pAol 
rapof ti ng program? 

B. Would you favor tha aatabllalimant of 
an A*SAP (A^ap) program, oppoaa K. or 
oaANar favor nor oppoaa K? 


JO20 Poaa your airina havt a Right Opai a tfo oa 
Quall^ Aaaurar>ca Program, aomatimaa 
canadFOQA(FOOua)? 

CLARinCAnON; Thia la a progr a m at aoma 
aMinoa that analycaa opar a tkmai data 
rou(lr>aly coHadad from tha fHgM data 
raoordara wRh corkcurrarKa and ovarolghl 
by tha piot*a union or aaaoctatlon at that 
aMna. 


A Would you favor tha aatabNahmant of a 
FOQA (FO Qua| program at your airtina. 
oppoaa h. or wafthar favor nor oppoaa? 


B. Hava you boon bdafad on tha program? 


C. Do you ballava that tha c onUdan U allty of 
POIM (PO Qua) data la adaquataly 
prolacliad? 

CLAMPICATION: ConfMafkttanty rafara to 
boltk tha Idantfly of ttia pilot fl^ng tha 
aircraft arkd to tha uaa of tfw data. 

D. Ara you awara of any aafaty 
Im p rovaman ta that hava raaubad from 
tho FOQA ffO Qua) program? 



• 











m.. ... 

_r 


w • 




4 




w . . 

4 MPTOJOM 

a 


pauQit. 


pcmca FAvcn NOAOPPCM 

9 


f 

rw a 


traarojooa 


a 

wa 


or 



• 

■m 

0 



tv 

t 



' 

.a 







m 


• 
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Wt T> rtfcf tf d m h— rtnq about P>> xrffy cufcw 1 youf AirtriA, — by youf — ft* 0 f 

man99tm»nt 9/ mn cf tfmnaotrmt. ^ m w tti» C 6 O . Dir<c>of <i> 8«Nty. V.P Drvdorof 

FIgM Opvaborw, and other aenor managamani 
ceo • CH£P eXECUTTVE OPFtCCK 
vf»*vcemsioeNT 


JOdO Ooaa your aMne hay* a C£X). mtoelofl 

at a tamawt ofi aaifaty? 

CEO ■ CMEF EXECUTWE OFFICER 


J031 Ooaa your aMna have a OIroGtor of Salaty? 


J092 Doaa your alrttna havo a V^. of Safafy? 
VPaVICCPREStOeMT 


J003 Hava you obaarvod a atroog comm Rm aht to 
aafaty aroong aanlor managaruanl? fTMa 
quaatlon focuuaa on behavior.) 








OK 















NO 


a 

VEi 










fMRtoxag 


VM 










A. la thia aenlor managamant 
cowro Ri fi an t to aafaty raflactad 
throughout tha or^anbatton? 


J034 If you have a safety concern, do you have a 

machanlam for bringing that concern to the 
aBanbon of aerdor ara n aga m ant? 

A. How affacthra la this machanlam in 
reaching aanlor man ag ament? Would 
you say <REAO CATEGORIES)? 


NO a 














... 



* 





NHatiOeiaoWa 
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»liflQna< Avwlion Optnbot m Montonr^ S^tvf Am Carrttr PM SufVtr (Vtr AC^July 1 5, 2003. v 002-004) P«g* 26 


SCCDOM O: QUESTIONNAJRC FKDBACK 
MTROOUCnON: 

I onijf Ntm ■ ooivlt mar* quMfton* and tfWM are about your fMcbont to MV«y w« hM )u«t ilOAt 


Hoar eerPM ara you M yoM aocurataly NoioenAMaiai 

oouraadaiofttiata4atyralaaad*v*n(sir«il fly^eyooa^ — 

aUtad you 00001 ? WoUU you aay you w«r*(KeAO Med a»Mly ccr iaaat 

oucsnoNS)? y>ycei*aw<-._ 


02. WOra any o> lha quaa t iona I a a fc ad confuaing. 
poorty ovOadl or ambiQuoua? 



A you p«aaaa oaa c r^i thaao ouaaooft prowarwa? PgCOAD vtHOATm ATCOMPtEDONOP 

MTtRVWW. CMTEP QUESDON NUMKft 



A wnai are viaaa proowTv ' 
WEOFV 


04 


Do yoo uaa 1h* ineamai at homo? 


W . 


.1 
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N«OonilAvipnQf<Op<fitWff»MoNlortnoS»r ^ A»C«rrttfWo<84^vty(V»AC’July IS. 2003 v 008-004) Pto»27 


06 Ooyouh»»^iwy0thycom>wnticf •u go»< ft c n t>bo»<»iit<^y/?RECO>W>VERai>TlM 


PANEL PA)$WONO HNIT 


NitOPAM Wp would B(« to b* «blo to Ink tho 
mtonnioon you givo UP MCh ttnt WO col. 
Bpcpupp wp do net Ink your inlornwo o n wth yci^ 
fipmp. wp wo^ 8(p to rpcerd on imftMOupl 
ppppwofd cpn upp lo ink your OPtP Mpywt 

pIcippptwwppppppwonithPtyouwdIrppipttoMp 
w« cNI you •gan'^ 

PCKPAM RCCOAOPASSiMOAO 

ASKPONMMT P ipup gbp up # q u pp t ipn thpi wp 
can UP* pp a hktt in cap# you pro unablp to 
nmpwfearycuf pa pp wcfdtnanaitinnawocpl 
For tnplanca. i you choopa tha worO *AEO* a* 
your ppapwonl, you haE quppten eoPd ba IMial 
ip my l*«ortp ecdorT* 

PA8TFATM Al 9tp and cl your M mipivipw you 
gmro up a ponaword po wo oouM Ink you 
irdormation acropp quarlpn You hM quaabonp 
WOP (MNroutSTiOPf) Whai wap your pappwonf? 
UCOAO 

REPEATPAM: RECORD PAS6MORO 

IFPASEHUOROMOTINPASSairOROUST T>WWO«d 
you gavo mp dots not mpich ou M of 
ppppwordp. Pprhapp I toadpd I wfon^ Mow do 
you ippi you pP oW crd? RETURN TO 
repcatpass field and record password 
AOAM IF WORD snuoofsirr MATCH ATTER 
TWO attempts, cuck. suppress 

SUSSPASS Sirrop (you cani rampfnbar/wo donl 
pppm to hp^) you piowoup pppraord. wa‘d 0i* 
you lo choopa anodwr paaaaord and NM PO wo 
tan Ink you fUluo imprWPwp Maywoplaaaa 
bfv* prvOtpr paopword and hnl that you wR 
rppaal to up wban wo oal again? 


tNisa*fnwMiwsiitpipECOPAr4PM«i i^otpor 
LATCR OTP aUT NMP COM^XTCO PITOIVICWt on PAST 
PATH Y* 01P OP LATtP vaio ParwOUOLY owwt 

PAaawopOL 



TMCa PFTCPWMSP TO ocaa 
aacoRONBFr 



F aUCCEaSFUU TMC8 PfTBIVIEWBI TO BCaiT 
0 PL d PP f t a cfl , tAlgp PKTIPVMWPt TO P u e PPAaS 
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HtBon<tAv<tttonOp»f«tentMcri<ortng8tr^ AH Cifr» Plot Survty (VfACsKity 15. 2003 v 002-004) 


KMOINT AgtirvtfwnkyowwrymuehfDryourimtantfyourhtipiMOtVMtturvfy.Vowrinpulwthtfptt^MOQn 
wdulry m gnwt dmi lo w — ur» th» ol w ility in lh> < >1 —0 11 tytttm and b> h«td in confWtono 

W PANEL MCMKft Wt’R bt CMtktg agtin in tfWM meNTit lor your (2**/3*M) inoofvixr. 


QUESDONNAIRE LBIOTH 


ou Mn owANA O tjNOTW onirrm Mil 
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Pag* 

81 

H«ve you nown M a eommerbei pM Ainrto ote M 

60 d«!^7 PteoM Oo not mciuOo rton-commofoal 
ftgM Umo lucMa mMwy or racTMtbn^ lyev 




vaa 

fCONTtaJK) 






SU 

Wm any of th« eemnrtorool oonducM M an 

air eamar pilot or oopnot ftyvig undar FAR Pan 121? 

Wm any of tha oommarooi flying oortdudad at a 
pflot or oopflol flying a fkaad «f*ig ■acrafl lor or tad or 
othar oparattona ur^ FAR Part 1 367 




vtfl~ .. 

lOOOf flr^i.ASKflfil 







vtfl 

- <0008 84^. TO MQ 
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During tha laat 60 Oayt, (fld you fly a ItcaOatng 
arplana aa a cRflian. norv4onvnarQM, ganant 
i iiitran pilot or copilot logging houra undar FAR Part 
917 












NOTCTOI^rTERVIE1AER COPAOT lOOQINO HOURS 
UEAKS THtT SniE FLEW AS A COPfl^T AMD tOOOCO 
HOURS M laSnCR OFFIClAl FAA IO0600IC 




83 

Ourmg tfia laat 60 daya. hava you flown aa a piot or 

Uft 

(OOMTiaje) 
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FUOHT MOO€ CHfCKPOflfTS 












88 

SELECT ROUTMO FOtXOW FIRST HlSmUCDON 

NOTajaai£.NoiMaiocoosoMS*- 
OOTOTERMaaaTKJNSCWFT.— _ 

...1 


aMU1VMONiAOUT€QM.V OMLY1 
njQMT HOot omctm. fouowocooo 

noun 1 


euoau M mo OT Mom nouto 
CHKK M TMO OM MOf« ruOKT MOOta 
IIAM)OIB.V ASSOI ROUTE . . 9 



D^mr AKfiUl^UBtfT 

IKTCTVgVr 

« 

89 

w^Uic AaoiuPWiKn 1 

IF 8»«3 ROUTE RANDOMLY ASSMMEO 
ELECTyiONIC 4 U.V 

OAaiTBIVICW.AmAMEROUIl 

Oa a<TOI^glW.>CJOQPTER ROUTE 

a 

a 


CONTaajf MTH tCrCOUUNO arTERVM 



TEMMiATKM 8CRI^. (Vn tony. bul your rtotn nQht Oom noi mm out 

fRqurwwntt Tharfc you ter your — ft tn c R 
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QtrwXAvtattonSurvy 



SECTION A: BACKOAOUNO QUESTIONS 

ifTiHIOUCTiON: nwQW^ON wAhiltw about yOMfQtoifil%ing<QS>Ni»nc» 


OA1 Do you hoN wn AT P ciU ft cf or tnNrum>n t fitlnfl? 
ATP«AlfUC TAANSPOfTT PILOT 


A Aroyoul^A curom? 

rS • MSTPUMCNT PUOHT PULCS 


OA2 During your 9h, how mony houn In 

lolaJ havo you Rown M ■ plor? IndUdo ol typat of 
tying ndudng FMI Pari 121 aP carriar oparalioni. 
air taad or ottiar oparabona unSar FAR Part 1 36. 
ganar a l a wiaoon ftinrig unbar FAR Part 91. at ««S at 
mMary tan«a and UMghI tying 


NO.. ... 

4N9T0A9 

a 

RF 

IMP TO 

t 







Mt 



ftf 










OK 
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OtncralA^MonSun^ 


P»9»2 


MCmOOUCnON: Th« rtM of tfit QuMMTtt «« ftitr to 
your flytng ooiportortoo during tho iMl 60 doyt pnor lo 
today VllhwMr I tsy tfw ’M 60 I am rotonng 
to tha panod from (START DATE) througfi CENO OATC) 
Alio, for ai lhaia quatUor^. I ail bt Mkit^ you about 
avanta aaran you flaar aa a plot >n com ma nd Of copdot 
logovig houra In your officai FAA logboofc undar FAR 
Part 121. Part 1S6or Part 61 Fm I aeuW Aa to aak a 
law quaattona about to# lypa of flying you ht ¥0 dona to 
tooMOOdaya 


OAl OurtoQ (ha M 60 daya. hoar many houra rM you 
•y aa a pKol or oopttot undar FAR Pan 121 . Part 
136. or Part 917 

00 NOT INCLUDE URJTARV OR ULTRAUQMT 
PLYING HERE. 

A. rdjuatNtotovartfy Youaanfyouflaar 

(• A3) hovra during 9ia laai 60 daya laMa 

corral 


B Outog the laat 60 daya, hoar many houra 
did you ly? 

AS A PHOT OR COPILOT IMOCR FAR PART 
121. PART 136 OR PART 61 


TOTM. HOURS aOMN LAST 00 0AV8 1 l-i. 

NO HOURS TB1MMATE anCRVieM. OQCC aiOT 
RIOWF' 

P HOURS M AI AM AaOVC MS ASK A 
OTHER MSI owicmoa TO A4 


NO. . 

vn 

. .<A»( 6 )... 



OK 

.fWOPTOAd 


I I i 


» 


OAA PSA DOCS NOT at, Stop TO GAB. 

Hoar msny of Chaaa (a AS OR Ase) holpa did you fly 
aa an airpiana piM or ecpM undar FAR Part 131 
air earrtar oparraclona? 





66 

» 
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HOURS CAMNOT tXCCeO MOUM ai AITAIS 


FAR PART 121 REGULATIONS GOVERN THE 8CHEOULB) FUQMT OPERATIONS OF COMMERCIAL AAUNCS 
AND AIR CARRIERS FART tat COVERS LARGE JET OR PROPEUEA-ORiVOf AIRCRAFT IMTN A SEATNO 
CAPACnY OF MORE THAN 30 PASSENGERS AIOOR OVER 12 600 L6S 0R06S toClOHT 


OA8 Hoarmanyofthaaa(aA3 0RA3e)houra<Sdyoully rrmfif Vf hfn m I I I 

aaaploiorooploiforairtajdorachar aa . 66 

oparatfona un^ FAR Part 1367 OK ,,, 66 

HOURS CARROT BBCfD HOURS M ARAM aaiua A*, 
a H. ASK A OTHERS aiCUJOaiO aiT MO ais 
tnPTOAA 

FAR PART 139 RCOULAT10N8 GOVERN Tie FUOKT OPERATIONS OF THE SMALL (LESS THAN 12600 LSS 
0R06S WEIGHT) COMMUTER AdtUNES AND AJR TAXI SERVICES AU UN9CHE0UU0 PASSENGER OR 
CARGO OPERATIONS ARE COVERED UNDER PART 13S 

A. Of lha (6 AS) hours flopn undar Psn 135, how aroto 9 F/vtil 6 iNHH.i*NE L__L—_L 

manyooaaradtofhBad-wingairpianaiT rf .66 


HOURS CARNOT EXetES HOURS M AA 
a AM* AS STOP TOM. 

a <AA aar OR aiA ASK Ais 
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Ofifil Avuaon Sufvy 

B orth«< 0 A 5 )heur«io«(nundtrPvt 136 .ho<ir 
many oocuffid In h »c o p Hn? 


PiptS 

9 HOUM FM mtCUCOPTBI I j I 

m — ^ — _• 

OK • 

HOUM CMliOT EXCCH) NOmtt M M MMIN AaA. 
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Ati^jPQft Si^vty 


Pl0t4 


<W« HoMrmanyo^tiM«(«A3 0RA3B)hour»ddyou •HOuittUNO€KFMti I ^ ^ ^ 

fljrMaffWMTiltvtillonploloroeplolt^idtr — - ■ ■ 

FARPwtOI? » ••• 

NOUM CANNOT eXCCEO HOUm M AtlAM IMUt «UM 

» AM A. Ot>«mL (MCLUOMO W7 AND Mt. 
•lOaTOAT 


MFAItT»lREOUtATIOM8COV^ BASIC AMD OEfCRAL RULES FOA Ml AACfUfTO^OUTIONS PART 
SI OOVCRNS THE OPERATION OF BUSMESS AIRCRAFT ANO AIRCRAFT USED BY OEICRAL A\AAT10N 
PROTS MNO FLY FOR PlEASURE 


A. Oftha(BAB)hour»flo*Antf>tfafPsrt91. hoN 
many occurrad M fbiaS-iainQ aaptanoaT 


B OfB)a(SA8)h(Hnflo«MiiNidarParl91. how 
many occurad in hticoplafi? 


• HOURS FAR 91 AMPUMi 
P» 

» 


■I I IJ 


HOURS CANNOT Excns HOURS m AS 
RASA* AS. SKIP TO AT. 

» «AS. ttr OR MS. AJK ASS 


S HOURS FAR tllCUOOFraR 

Rf 

OK 


-I I I I 


HOURS CANNOT tXCtCD HOURS n At MKUS ASA. 


MTROOUCnON: Now I'd Rm to ask a law quasBons about lha nuwOar of lakaoAs or fHoMs you mada Airtng 
VwlaateOdays Mb uaa Via Mrmsn^rrthrougbouttNslnlaFVtawlomaanBia period o<tniabatafaaoaaeh 
Moooff wid Isndrtg. atari t that tuna is abort sueb as ter minictDra taacMng ttudofSA (o land ard *teueb and 
900s.* READ ATAtt MHCN AmiCABLE 

TOUCH ANO GOES • ^ARV SHORT FUOHTS MMEN PRACnCaNO TAKEOFFS ANO LAN04N0S 


OA7. rA4*S.7,ORS.SIOrrTDAS »A4>S.REAO 
During tba (S A4) hours you flaw as an airplaoa 
pilot or eopBot taidar FAR Part 121. how many 
Mnct lUgM laga dU you aa^pananoaT 


OAS. 9 ASA ■ 0, 7. OR a. 8»V TO AA V ASA > A RtAO 
Ounng tha (d ASA) hours you flaw at an airplana 
pBolorcopBotundarFAR Part 19$. how marry 
takoofls ddyou aBvananoa? 


A (forhoNmanyofthata(BA6)ftgbls/For 
thte fiteht) ware you Bio pBol in command? 


B (How many of thaaa (i AB) nghta 

ocoNTOdOid thia (Ighi occur) aahar as or in 
pan during nighttima oondBons? 


a OF i£OWTAI«OFFS PART U1 I I I J 


RF sar 



aia 



as? 


—SSI 

P AS SLANK. A MT OR 9M. SHIP TO AS. 


a PART ISSAIRPUUCPUXITM COMMAND 1 1 J J 

or _ 187 


- SOS 

Muar as EQUAL TO OR tsas THAN AS. 



1 1 1 


mt 

W- - - . 

m 


IRIST M eOUAL TO OR LESS THAN AS 



OfUfl Aviidon SuTvgy 


C. (How many t0 ttiaae (• Ai) flQMs 

ooeurrattDid IM AgN oecuO undar an 
I.FR ttgMpIvi? 

IFR • aiSTRLMCKT mOHT PMt 


0. (Howrn«nyorihaM(«A8)lkQhls 

wa<%Wa$ RqM) so nauocal mAM or 

mora in Ivtglh'^ 


E. (Hoarmanyof Aiaaa(AA8)tfior«» 
wamAMaa n» Aghi) to or Aom 
vAamabonai daa&naoona olhar lAan 
Canada? 


QA*. r AIS«0.7.0«tAS»aFTOA1ARASg>Q, 

RCAO: Oving lha <• ASB> Itourt you tmt aa a 
AaUcoplar pttol or copAol undar FAR Part 
1 W. how many takaoffa M you aa^ananoa? 


K (Far tMw many olthaaa(fA9)aohta^or 
vaa IlgN) wara you tha pdot in oornmand? 


B |>towfnanyofthaoa(BAS)ttQhla 

oooirradOid thta fhgM oeci^ aATtar ai or 
In part dudng ntQNUma oondfaona? 


C (Howmanyolthaaa(#A9)Aghto 

ooajnadilM a^ht 00 ^ undir an 
LF.RffgNptan? 

IFR • MarMUMCMT rtjOHT MUlfS 

D (Ho«vmanyorthaaa(dA9)Agrai 

warWWaa thto fl«N) SO naudcai mAaa or 
mora in langlh? 


B. (Mow many of toeaa Qi AB) fUghta 

wam/Waa PM fiphO to or from 
Mamationai daaanaiona olhar than 
Canada? 


0A1A PAiA«Oi7«ORAiKIFTOA11.PAaA>A 

ttAO: DudnQ toa (d MA) houra you Aawr aa an 
airpiana pitot or copBoc undar FAR Part t1. 
hoar many takaoRt dU you aitoohanoa? 



a A10 aLMot. A tar oa OM. aroF TO A«i 
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GfioH AvtaPon Sgrvty 


PiQte 


A. (fpf hot many of tf>— • << A10) gg^t^or 

9M» agN) wprt you th« piet <n oofrmantf? 


• mm 01 A0VI/MI P«OT M COMMMO 

tv . 

OK 


,U_L 


HUtT M MIMI. TO OH l.nO TXMI AIO. 


B. (How many of OwM (i A10) Aghto 

ooctfradM this flighi oocuO MNv tl or 
In pan Ounng nigmoma condHtona7 


C. <HoMr marry of V)aaa(iAlO)AgM» 
occkAiwkDd (Mi fhght occur} (^idar an 
l/lt Agmclan? 

PB • oarmMBfT FuoKT nuiss 


• NMTtl A«0>UUCn.K>rT8ia0KT ^ 1 i 

fv - - .«a7 

OK m 

awtr PI aouAi TO OH ma THAN Aia 

• nurTfiM«>i>Mcru^m>N 1 i 


HOST ae eouAL TO OH Lfta nuM Aia 


O. <Moia many of »aaa <i AlO} ftghla 

warwWM 0it» agN) 50 naubcal miaa or 
mera m langih? 


E. <HpamanyofVtaaa<iA10)flighi» 
waraVMi M» BgN} 10 or 
■Oamaeonal Oaiarr^oni othar than 

Crada? 


• MHT»iMm>iMcax»rrauMa. ^ ^ t ^ 

RF m 

OH «M 

■uar aa aouAi. TO OH uaa THU) Aia 

aPKHToi arraHWATiONM. ^ ^ I ^ 

m IPT 

on m 

Huar 08 aouAi TO OH uaa THAN Aia 


0A11. r AM«Q.T.0BAtKlFTOA1Z.PAIB»0^ 
UAD; Oumg tha <f AOB) houra you Haw at a 
h a Wc op ta r pQol or copAol undar FAB Pan tl. 
hoar many lakaoIRa dU you axpartanoaT 


A (forhcarmanyofthaaa(aAII)agfti^or 
taa SghQ wara you tha pU In command? 


B. (Hoar many of thaaa {• At 1) Wghta 

ooctfradtCid tMa flight occur) aihar al or 
la pan dunng nighBlma comSlona? 


C- (How marry of thaaa (t A1 f) Aghta 
occurradOld (Na occur) undar an 

If R fhgN plwr? 

PH ■ a«THUMENT FUOKT RULES 


0 (Hoarmanyof9taaa{•A1t)fl^}hfea 

aiaraAV ai M» fighi) SO nauOeal ndoa or 
mcra in langti? 


opwiritH&icopTiiirMaoFn. I I I I 

RF m 


a Ai 1 auuM. a tar OH aia amp TO Ai 2. 


• FtmaiPLOTMCOWMAMD ..1_L-LJ 

m m 

OK . — m 

auar ac aouAL TO OH Liaa THAN Alt. 


aFMTtirCUCOPTBIFUOHTSiaQMr ^ ^ ^ ^ 

RF tar 

OK . taa 

autT ae eOUAL TO OH Lfta THM Alt. 


• MmairejooPTCRruoHTFUM 



Muar at eouAL TO OH Ltaa THM Alt. 


tPMITairejCOPTBtnJGHTatONO ^ ^ ^ ^ 

HF .... m 

0K..„ — taa 


MUST at EOUAl. TO OH Ltaa THAN Alt. 
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G>nfi< A>i<«Oon Survy 


thif 10 or fcvn 
imofnoOonol dunobont othor flow 
Canada? 


iPMrrtt Meucoma w ium o CT OWH. .1 1 .1 I 

w m 

OR Mi 

■uar at touAL TO oa Ksa nuM Ai t. 


INTRODUCTION: 

Earfaar you Mn^ ca Md that diJwgthalMt 60 ^ya you ftaw CASA»A6A Oft AS8»A68)h<H<r»ae (an arplani 
a haioop l ar) piol er oopiot ItyInQ undar FAB (Part 13&Part9l/Part 135 and Part 91) ForOwraatof lha 
Marviaw. I wO ba aaUnQ you at>oul your a ap anan ca a Oylno (airpianaa/haicoplara) diatng eiN panod of titna. 


OA11 lamnowQoinoiofaadaMtof^ffaranltypaaof 
Oanaral avtaOon iyv^. Piaaaa Mi ma tf Oioaa 
(• A5A«AAA OR ASa^AOO) hoin awoNad any ol 
thaaatypaaorOyino CM you undartaka any (MrplarM 
haioopiao fligiat (RCao CATCOoaocS)? 


A. tor rnght Inatructton aa tha in a tru cto f? 

NOTE NCLUOCSCMlCKOUriVOfTI 
8. tef rtIgM iiitmeatoti — Utm atuOmnn 

HOTE NCiUO€5 CHECKOUT 

C. tor corporala tranaportallon aa a pdot 

amoto^oCacorporMaRoWdap a rtwanr? 

MOTE: ooet MOT iCUIOC QMITBI ruarrt 

D. aa part of your own buainaaaac6vltiaa7^_^ 

E. tor oovammani or pabilc pu r poaa a in aacraft 

owmad or oparaiad by Qovammani anottaai. 
tcmaumaa calad pu^ uao fipWa?..— 

F. MR piying paaaarigar i . atootowan aa rauan u a 
paaaanpan? 

Q. tor cargo or fraight tranaportabon Mhout any 
payrtg paaaangara? 

H. torlranaportlngpaatanlaorcrltlealmatflcal 
product! aucfi aa OTBana tor tranopam or Hood? . 

L tor ra cr aai t ten or paraonal tranaportation net 

iiiOfraHil nth 

J. tor any otRar pupoaa? — 


1. WhalwaalRapurpoaa?8PEOPYVER0ATat 


NQ_66iii _ W _PR 


ApproDdmaaaiy bow many 
boura would you aay waa 
dawotad 10 (BOLD WORDS | 
WAOao)? 
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Qfwfil AwBor> Stfvy 




0A19 Farth*(tASA*AftAORA5B»A6B}teutiyou«««M(«nai5plM'»h«lioQpltr)p«ol 
orooptkilindtr FAR(Part iM^artOliPart ISSand Ptil^) in ifw laat 60 day*, 
plaaMlalmaaiorir>a<iirpl«i*'he(icop(«Off«Ma8ndfnodatiyouAaw RSCORO 
VmATVi Ll$TAaa«00a.ST>CNASKCOUIMM$AAN06FOII6ACH 


ASK eoujm e oia V V 
«0 aAMMnofi maicm 
« enen ooiMi tenau 


A. 


C. 


■AKEMODCL OOeNTTV PROM DROP- 
DOWN LIST. PHOT ON UBT. RECORD 
VERSATRD 


Oiprio tha laN 60 
day*, hew many 
hour* did you fly iha 
fUAKE/MOOEL)? 
HOURS SHOUIO 
CCXMLSUMOFAM 
»ASA.ORA86« 


How many anQinaa 
deaa tNa NrtraS 
im? 


la Vila an 

axpanmareai 

airpiana^ 


AM 


no YU HP OK 
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PbQbl 


SeCDON B: BAFETY RELATED EVENTS 


INTRODUCTION: 

My n«rt Nt of »i>i fcnt m9 aboiA uMy rMMd •wwKt Juot « a mrtmdm, fd IM yog to raipofi orly •vonts tool you 
oaportowood flying undor FAR (Part iSfl^Part fti^art 135 and Part 91) on (an aNplanafa hoBcoptoQ on wtdcti you 
warvapiolorcopioi Tha ton quaaoo ra art iboui aRuM w Ht fala to d avanto. 


OOH Ho«vfnanyt«TwadtrlngihataalSOdayadM aaouaitoNT pwotociia t M I 

(an torptonaf a hMcoptoQ on aMeh you wara 

aploioroopiotdNarttoanaiamotoakport PAiaFTOtat. 

Of fatow to land baca ua a of an aireiNI 
apulpmax pfoOtomT 

A ASA OM.YP MORI THAN ONE MAfCtoWOOEL ngCOWOIWEai006LafR0MAia. 

IN A19 \M)lcP (ipl an a^ia l copto O 
axpo^Nnoad (hit aoulpfnant pfoblan) (moat 
laotnlly)? Wm I (READ AM MAfCAHOOtL 
UST)? 

B )AINN natamt eauaad tha (moat raoanf) ■p ec ty 

duaraion ct latum to NndY 


OCR2-A AIRPLANI CNLY 

lamgotogtoraodalitofpaaitotoaaytonimNAmeeonaortaMa Foraaehona. ptoatatolmahow 
many amaa durtng Oto laN SO daya an IMBgM airplana on artatfi you aafa a plot or ceptot aaptftoftoad 
anyoftoaaamalfunctton a or to aufaa. EaptooaofatoJpmantdoatnotapply, ptoaaaanawar'not 
appleaMt* ratoar than *nro * How many bmaa dd you aoparianoa (READ QUESTIONS) 


COl.k 



Ol OidyouaMpartonotamaSuncaonorIliufaof 
any othar wrcrtol danoa or tyatom dunng tot 
lattSOdayt? 


YW 1 








OR 



HA 

m^romm — ^ — 

t 
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Airtgtion Swvty 


Pag# 2 


1. ASK OMLYF MORE THAMONi RECORD MMAMOEL# FROM AIS 

MAKEAAOOCLMAa WNch 
(MpMnaffwacoptar) «> 9 >ww>oa 0 thM 
RQuipfvwE preMtfn (fnotl f9C9f0f^ 

Wm I (READ A13 MAJCAIOOCL UST)? 

t VWcR DRyteR Of t y iMm WREuncSoftRd tfwcw i 

orMdfmoM rwanEy)? 


OEII2-M HELICOPTER ONLY. 

iRmgeR^QMfaadRNlofpo n fcli h RtcopMrmMhincflon R ortliuraa. For aach ooa. piMM M ma hoar 
many tanaa (Agv>g oa Mat 00 daya m iMPglR hall c oplar o« rPkR you atara a pSo< or oopiot awparianc a d 
any of ihaaa matfGnoinna or failwaa. If a pMca ol aquomarc doaa no( apply, plaaa# anawar 'not 
ippAeabla* ralhar vtan *xafo* How marry timaa dU you aovartanea (RCAO OUEsnoNS)? 


tf^SSroSSen 

AikONlY^yOM 

THAN ONE MAKEMOOCL 
MA 11 Which halocpiar 
arMhonoad thia 
malfunction or Mura 
<moai raeanay)? Wda K 
(REAOAiSUAKEAllOOfL 
UST)? RECORD MAKE/ 
MODEL a mOM All. 

B ftlunolfn»nn»t»nn»ctm»7 1 1 1 1 

C FaiuraoflhalanclInggnrioaMndortatracr*. 1 1 1 1 

0. TairelwMiim? | | 1 1 





E Famra of Era hydraulc ayaiam? 1 1 1 1 

F ValMIranamlaaion warning of peaarRal Mura? ■« III! 




0. Did you aoMahanca a maMmetton Of Mura of 

TEi 



1 

arry othar aNcraR davioa or ayaeam dufino fra 

RF.— 

— ..MWPTOOWR , 

— 

Mao<ta|n7 

OR- 




MA 

.4MCR1DRM 



t. ASK ONLY V MORE THAN ONE RECORD MAKEMOOfi. a raOM At» 

MAK&WOOCL M AtS Which 
(aapianafhaRooplar) ariparianoad RMi 
aQMpmant proMam (moal focanlly)? 
waa I (READ A1S MAKEAIOOEL LIST)? 


2- WhkROmRoaorayatafnmalh.irtelionad 
or Mad (meal raeimOy)? 
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OintrlAiiliCcn Sutvy 




Hour nwiy miM tM( 00 dW 

oopiol d^wnsno* vncit*. irw. or fuRiM mat 


ht (airplan^wlooptar) on wtUcf) you «fart a plot or 
ongtnM in (AM QUESTIONS)? 


ma anQda. aoQata ooinpaitmani or 
»(nim<saL)?« 


maeoc*pt? 

ma cargo or boggoga oroo?^ 


ma paaaangar oomporSnant oraa? 

■oma ptna omar than In the angna or 
naoaba (mm SSi i ). oodipa. cargo 

MOO QfpMa^ngoromo? — 


• ocPdODicra 

IPKanaTO 

amni 

OOL. 1. 

'ASROMLYip'iaeiM~ 
THAN ONE MAKE! 
MODEL at A1E 
wad) (aapian^ 
haaoQp^(moal 
reoor«y) 

aiMartancad amoha. 
Ere or Amea tn (ERI 

A€)?MmI(REAO 
All MNCE/ MODEL 
UST)? NECORO 
MAKEROOOe.1 
EROMAIE 

E 

“(OTirw {f ERI A €J ' 

tmaatharawaa 
amctie. (Ire or 
fumaa in the (EM A> 
E). how many 
irrvoiyeQWma 
•moke, drear 
fumaa tmehie) 
eiacahcai 
oomponanta or 
winng? 

MM 


11 1 1 

MM 


MM 

.Mil 


1 11 1 

1 M 1 


U.IJ 

M 1 J 



spcaprv^«A£ 


OESA Ourvig me (aat 00 Oayt. how many mnaa <M an 

wm g ht (aaptrmmeaoopftar) on wtidi you ware a 
plot or oopiM eiperianoa a precaubanary engine 
•huldewn? 

A. ASK ONLY r taOWE THAM OME MAKEmOOG. 
m All. VWiich laapiarWhaAoofNao 
ai^ananced a praeausenary engine ahuidown 
<moot raoar«y)? Wm • (READ Ail 
MAIgAIOOCL UST)? 

OEM. OuimgmeiMte00ny«.howmanyamaididan 
aOlghi (oirpiviameicopair) on wWeh you ware a 
ploC or fopAp t aapenence a tout a**gp^ laaure? 

A. ASK ONLY m MORE THAN ONE MAKEMOOCL 
m All. WWch (airptanar haScopaar) 
aaparienced s loW engine taSure (moat 
raoanOy)? MSa I (READ AH iMKEAtOOEL 

usr)7 

OEM. During the loot 00 days, how many emaa<M an 

HIghi (aaplwdihelc i Dptar) on which you ware a 
plot or copiot aipanance total loaa ol eiadricai 
powar? 


aanECAUhONAMEiKiacftiimioviaN.-- L-LJ-J 

ma.MPTDoaM. 

HECOaOiaHAEiiOCCiafWOMAH 


aroTMiMOacfAiuai Mil 

waaneTooiM 

ascoaDiM«Aaooci.afwaHAii 


aTOTM.B.£CTRlC«CFM.unE . MM 

W 0.0100 TO OERT. 



G«nfil AvBon Swv^ 


OER7. 


otnt. 


oem. 


OCftl*. 


oont. 


OCRIl. 


A ASK ONLY r MOM THAN ONE IMXEMiOOCL 
INA1S (orpltnMf fMSeo0sr) 

taqwfMnced • tolN loss ol e M rl ri rM powNr 
(moN rsosntfy)? Wm I (NCAD At) 
MAKEAIOOCLUST)? 

Dufmg Vm MN 90 Scyt «MMn you WN9 pM or 
copiot. how many tMTwodU you docovor9wt(«n 
•irp4M>oo hoSeepNr) hso meorrocl or bogus ports 

msMSod? 

A. ASK ONLY# MOM THAN ONE UAKENOOCL 
MA 11 Vttuch (arpMno/ hsiccptor) hod 
io co rrset or bogus ports swtiSsd (moN 
roeorNy)? WM K (READ A1S IMKEASOOEL 
LIST)? 

[How rnony hfOM <fMl you docovor thoQ Cobin 
doors, boggogo doors or oowbngs oportod 
modvoftorrOy dtfwg ngMT 

K ASK ONLY M MORI than OM MAKEMOOCL 
m Ats. WNch (MpMnM’ hsicopiBf) hod doors 
or oowtrv* * ^ Md¥or1 o nfly durtf>g MgW 
(motf rocondy)? Wm i (READ A1S 
IAM(ElMOOCLUST)T 

p4ow morty tirnM M you dMoouor PM] A door or 
wr4ow coTTM Oil 9M McraS ohM SI flight? 


A. ASKONLYlFMOMTHIWONCIMKEAlOOei 
IN A1S> MNch (oMpMrw/ holoopsor) hod doors 
or windows oorm off whAs M fight (moot 
rsconlfy)? WM I (R£AO At9 MAKEASOOCL 
LIST}? 

|How many MhM did you| ojponanco o cvgo shA 
or cargo oOTTMg looM? 


A ASK ONLY M MOM THAN ONI MAKerMOOCL 
•I A1S. WNch (oirplarMr hoioopMr) 
oi^onarKOd a cargo shA or cargo ooming 
toOM (most raoantty)? Wm • (READ A13 
HAKEMOOtL LIST)? 

Ounng Iha MM 00 days, how many bmM did (an 
aspMnafts hMcopstr) on which you wars a pdot or 
co^Syof aSarrptioSy wdMuai coriamlnatod 
bywaior? 

A ASK OSa.YP MOM THAN ONI MAKIiMOOCL 
IN A19. Which (asplana/ hoiieeptar) hod woior* 
contomSMod fuN (moot raoandy)? WM R 
(REM) A19 MAKEAIOOEL LIST)? 

(How mony tsnM did you| Sy or Ntampt to Oy wMi 
Ota oning typo of fuel? 


aCCOOeiSMEAlOOeLSFROUAIS 

STOTA; rMITS 

MM 

PAMUOTOOtRS. 

aSCOMtNKMBOOCLSrHOMAtS 

S TOTAl. BOOKS O^YN . 

Mil 

PKSNPtOOMISi 

MCOhOUAXEAlOOELfMOUAiy 

aTTWAJ IWWMm 

Mil 

PS.aiQOTOO«IHSi 

WOOnOtSMSMOOELSnSOMAIS 

a TOTM CfYTOO 

MM 

PS.MPTOdCR11. 

MOOROMMCMOOGLimOMAty 

•mTAj <vwTAiiBuimnBi 

Mil 

psiaiapTOotmt 

Necoooiou(&Mooe.irROMAiy . - 

• rOTACWNONaFUB.. 

1 1 1 1 


MS.iiQnTgoams. 
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AviaOoo Sorv»y 

A. ASK ONLY r MORE THAN ONE MAKE^MOOCL 
M All. VWiicti (MrpMn*' hAiccpMf) Amt or 
•MfnptAd 1o Ay th« ««or^ typ« of ImI 
(moot r«o«ntfy)^ Wm ■ (READ A13 
MMCEMOOEL UST)7 

OCNil. (How many tvim OM you) ei^Mnanoa ■ fMuro of 
9tt MMudo md teo lof or orMcial hortzort? 


A. ASK ONLY # MONK THAN ONI MAKMiOOCL 
MA19 WNcA (anpianM^ ftaioeplar) 
c^artartoad 9m ftiiura (moat racarOy)? Waa 
I (READ At3 MAK&WOOEL U8T)7 

B (Ofthat*CB13]timaa9iaatmudaMcalor 
(Mad. hoar marry occurradrOW 9m (Mao of 
(ha atttudeftfcatar occur) 111 a M njnaid 
mmtontOQicU c orrdBo na or LM.C7 LM.C. 

maana fra vMMty waa laaa than thraa miaa 
arrd^ tha oasng waa laaa tfran 1.000 faai 
aOova ground 


INTNOOUCnoat: 

My HaM quaadorra raMla 10 turlwlanea 


OTU1. During tha laal 00 days, hour marry Mnaa Md (an 
aapianaM haAoopiar) on which you wara a pioi or 
copdot anoomiar aawara ttfOiAanoa (hat cauaad 
largo f^vp* changaa in oMuda. aKapaad. or 
aSEuda? 

A <Oftho[«njlJaamturtHMnoa 

anoountan. hew marry ooourradOid (ha 
aauara itfthianea aneowAar occur) M UiC. 
oondMona? 

IMC ■ (NSTHUUENT METEONOLOOICAL 
OONCMTIONS 

B (Oftra^TUIIaaworaturtKdanoa 

anoounlara, how n^arry ooourradOld (fra 
aauara tutxdanoa aneouriar occur) In daar 
air? 

C <Or»ra[»T\l11aavarati^buanoa 
anoountan. how many raauEad/Did tha 
aavara (utKOanoa anooumer roaul) a cna or 
mcra oocuparaa haing ayurad? 

OYVI. (During tha laat 60 daya. how many timaaMd you) 
Encourdar waha lurhulanca (hat raauoad In 45 cr 
mora dagraaa of BveraS rot? 


5 
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G«n«r»l AviiBon 84«v«y 


Otftno9>tlactaO<Sty». ho«rmany»nM<lid(«n 
atplwWa halnsplar) on which you won • piM or 
CDpSot tack cccurpta wMlh6f i^iofnubohwhcn 

you hooooa i wMO okOomo? 


A. (Offw|f^l]timMwhonyautadiod 
oocufota woolho^ Monnioon, how wtfff 
krvotaod horMJ.8 okpont or centroAm?/ 
Ota tita ifTW whon you tacAoO oocunMo 


airport or oonboftarT) 


8. (0(tta|iWEt]timaawh*nyeutacfcad 
aocurtaa watahar inte nr ra Oon. how many 
trrvohfwd A>T)$ (A4a>?/Did thii tima whan 
you tackod ac curate 'awather rtofTtw h orr 
kwotvoA-TiS (A«a)7) 

AnS-Al/TOMATtC TtAMlNAL a#OAMAT)OW 
SYSTm 


C. (Oflha|iVWE1]timaawhaoyou 
tacfcad aw^ate woatfiar intcirmaoen. 
how many Inuotead a Flishl Sarvtaa 
8ta0on>^ lha tana akten you 
tackad accurate wa ath ar tater m aho n 
krvoKa a FIIqM Samea StaoonT) 
FUOMT SCAVICC ATATION ALSO 
R&CARCOTOASF&$ 


0. (OlthalawEilomaowhanyouiackad 
accurate waalhar Moinwoon. how 
many (nvotead FMght Wateh? rOd Ms 
hma whan you taOcad accurate 
waat h ar miormabon Invotea Flgta 
WaKtiT) FUOKT ^TCH • PAST OF 
FS8 SYSTEM USED PRIMMOLY FOA 
PHOT REPORTS 

E. (Of that* WEiIttmaa whan you tadted 


many itwohead tfia Autemattc Waalhar 
Obaarvahon Sarwica or Aulomafic 
Surfaea Obaarvacton SarvtaaT/Ota 9m 
tana whan you tediad ac cura te 
waa th armtormaBon rrvotaa tha 
Atao m ah c WaMhar Obaarvabcn 
Sarvica or Autemaoc Surlaca 
ObaarvaOon SarAcaT) 

AUTOMATA WEATHER OSSERVATtON 
SERViCE ALSO REFERRED TO AS 
AW 0 8 AUTOMATIC SURFACE 
06SERVATI0N SERViCC ALSO 
REFERRED TO AS AS.0 S 


aiACk w: 
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QWUWL 

AatnAM ONLY. 


1 1 1 1 


[How marry amaa <M you] dhrart to an 
aiamata aiM bacauaa of waMhar> 

MtoUMI ao TO WaSA. 


owta<H 

HtUCOPTER ONLY. 


1111 


[How marry Wnaa did you] dNart to an NIamato 
aMald. hatport or land bacauaa of waatoar? 

Mtojoorrtoi do TO NCSM 



AMPLAMi ONLY. 

[Hoa marry bmaa dW you] aaporianoa arrhama 
long that roduoad tia aircrafa abdly to rrrairdBto 
aMuda. apaad. NabMy. or dtrocborwi oortoof? 

aaxPcwwBWXAawwatwicam 

_I_LU 

OM»41 

HEUCOmn ONLY. 


1 1 1 1 


(How rrrarry amaa <M you] artoartonoa aaframa or 
reaor tong that roduoad Oto airerara obMy to 
mairrialri aUuda, ipaad, atobdiy, or daacional 
oordrof? 

NNfUWiaOTOllOS 


OIM4. 

(XatngtoaiaalOOdayt. how marry tonaa «d (an 
aaptortaTB holooplarl on amen you wara a plol 
or oopdM anoountor wirtdahaar or amicrobtffl 
oondMna that raaubad in an awapaad dawabon 
of 15 knots or graalar? 

atocoiarrtnwaioaid^Aaacnoauaar ^ 1 1 i 

NNOfurr aroa TO cni. iriucema ooMiaiut. 

OIMe»44 

HeUCOPTffI ONLY. 

[How rrrony amaa did you] aapartonoa teas of tal 
rcaor affacfvanaat duo to high danaty aMudo? 

a aoToa vftcTAMaaMT 

--I 1 1 1 

9tmUi 

ICUCOPTOtOfCY. 

[How marry amaa did you] aitoonanoo loss Of tai 
rdor offactNanaas dua to high asnds7 

taoToa cnsfCTs^caavMNM 

1 1 1 1 

cmrM 

HnjoorrtHONLY. 

[How marry timaa did yoi4 aapananca loaa of tha 
watola horttort dua to aduto out or brown out 
oortdNona on oMrar lakaoff or landing? 

aMMOMtOUTOOWmONO 

_LUJ 



®cn. OurtngtwiM 00 day*, how marry •PAxCHtnucr I I I I 

you diU Mtt id by o ppaaon jar wh*» w Itpht 
throu^ oomofiaOon or phyicol eerdaef? 

INaUDf 0 TiimiQ ON SHOmOfR 


■rmooucnoN: My naal quaaaow ara about aIrbofM c co fMct i, Jual at a ra mi wOaf. wa ara orty aMmg about 
avaritavraiyoua39ananeadth«iodcrt^Qnataa(OOOayauoOarFAR{PaniM^ortOl>Prt 1)5 
and Part 91 ) aa ( an aapMnara haaco 0 on p4o( or oopOot 


OAC1. How marry fimaaOd you aocparianoa a bird airto? 
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Gararaf A^abon Sufvay 


FagaS 

OACl [Howwanytimaadid you]Parfbrman araanraaoPon 

10 avoid an imminarc rnflighl eeffaion wffh anothar 
areraff ffaf waa navar doaar ffian $00 laar? 


1 I I 1 



OAd [How many bmaa ttd you] Ei^ananoa iaaa ffian SCO 

laaf of aagiaraoon ffom anofhar avcrall wMa boffi 

aaoraff warn aatoma? 

ouBanuunoararraVAiuTioN 

LUJ 

1 ■fTKOOUCnON- Tha nad taw quaanona are about groaad aparattana. 1 


ooei. Dicing 0«iMt 90 dvyv.twwmwiyllmM did («t • mvo MNCator. I I I 1 

■irptonM halcogitr) on wMc^ you «iv» • pM or 
oopM lond • tocaoon wthout 0 wind Md(. «flnd 
van*, or otfiar wa^ ndi ci of 9»fcd7 


OOtz. [HtwmnirlmMMyeulTalMcff.orillanvIloMa ,noncnitemM Mil 

cH urn comm locks, psoi artn. a ctut praascM 
gsai sia lOsckod 10 IDs skcnA? 

mcuioc* MT HOT UMTTB) TO OEM PIACS. 

EHOSC. OITAKE. OR EXHAUST PUIO*. TK-OOVTNE 


oocx gHow>rionyUTios<Myou|Esps>isnosanikol«<nod ssucTEOTWEan^— 1— 1— kajeopnRanPTOOtii 

aoonsd or ntocM triuoff? 


00C4-A. smAic ONLY Dufkieniolottw Osys. Hon 
many dmoo dW on orplano on wtaeh you wwo 
a pM or copiot go off tho odga of 0 fxnwoy or 
taidwaywtiaatfedy>o 7 


OQU-A. AiVLAMtOMLY 

(How many Inwa M you] Go off tfw of a 

rtfiway wtiia MUng off or Undmg? 


OOCdnA MRPLAMtOI«.Y 

[How many wnM did you) Go off lha orKf of tha 
runway? 


OOCTWL Aifm>Mi ONLY Oumg tha last 60 daya. how ma 
anaa dd an airplana on which you wara a piol or 
oopaot moehartandy anfar aw a ca a mnway? 


aOOOrriDQtNUNMLVrTAXIlMLY I XX .J 


aooofrgwEOFauwiriiY ,,1111 


a 00 OTF SHE OF RUMMY 

Mil 

a IMTtfl ACrr^ RUMMY 

Mil 


|lW many flmaa «d you] tagm tiMcff wMa anotfiar 
aireraff oeeuaad or uraa oroaaing iha aama runway? 


a TMora noa MTM oocuPCD auwAWY 


IJ-L.J 
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O^riTit Aii<i6on Swy 


0QC»4L AJKPLAMCONLV 

(H9«r irany timM M you] Land wM* wwthar airenrt 
ooc u piad Of itm erooat n g th> —ma fumcay? 


ooen^ AIRPLANE ONLY 

94ow fnariy Omaa M you] HE or ceAdt ««i • n,a*M«y 
or taaaoy Ight? 


OOfill. Ourmo Eta M 60 Oaya. how many ttmaa <M (an 
aaplNtaHi haicoplaf) on whkti you aara a pEol or 
00 ^ Nl a tfaar or oEtar ananai othar Eton a btrtf? 


OOI12A AIRPLANE ORLY 

flow many amaa dfd you] CoMa or naaoy ooMa 
«Hti a grei^ vahtcia'^ 

K (OfEia(aO£12IeoRaionaornaareoaaiona 
a«h a vound vaMdo how marry oocurradOid 
Eda coRaon or naarcoRaion wEh a 0 round 
vaMda ooour) wHIa yoi^ aPoiR waa on tha 
ramp or apron? 

B (Of1tra|iOC12]coRataniornaarooRaionao«Er 
a flrotmd vaMda. how many ocounadiOd tai 
ooRNon or naar ooRaion wth a grourrd vohida 
oeeur) whia your aacraft aaa on Era tROw«|<? 

C (OfBraIiOCl2]ooRntonaornaareoRatenawEh 
a ground voMda. how many oecurradM tiN 
ooRiion Of naar coRaion wEh a gmrd vahrda 
ooaa) oMa your drcraR waa on Era runway? 


OGC1MI MUCORTEROMLY 

^<ow many Emaa dM yor^ CoRda crnaa rt y ooRWa 
wan a gro^ vaNcit? 

A. (0> Era |E Qgl3] edhtorw or naw ecRiiCfa wEh 
a ground waNdt. how many oeeurradDid Etia 
coRaicn Of naar ooRiton wth a ground rmNda 
oocuf) wtrla your dreraA waa oparattng at an 
aapon. ao« a haiport? 

6 (OffiiiaOCl^oofldonaornaarcoRaantwth 
a ground valuda. how marry oocurradOU Eria 
oeRaor or rraar coRtion wEh a ground vaNda 
ooeur) ahda yotf aPcraE WM oparattng at a 
haipon? 

NOTATANAMPORT 

C. (OfEia|!EOE1^ooRa«naornaarooRaionawdr 
a groiArd uahlda. how many ooomadlOai Em 
coaion or mar cofcawn wth a ground va h da 
oeeuT) whia your aacraft uwa oparaing at an 
unpnparod iMng aaa? 


PagaE 
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Addition Sufvy 


00t14^ AmJMC OMLV 

OuclnQ tf« M 60 d^irt, how many timM (M an 
•ipdna on «Mtfi you nai# a plot or ccpiM naarV 
oi ^«nan c aagro<^ coaiicn o<hano»>afaroaft 
wtita both atcrafl twara on tw ground? 

A. (OfVia|iGC14|naarooBiiona«dihanetftar 
aaendC hoamany oc cu rradPd Ho naar 
ooOiion aOh anothar aacjai occur) oMa your 
atr ei ii waa on tha rarap or apron? 

B (Oftt<ay06l4|naarooat>onia<Hanothaf 
aaonA hoa marry oeeurtadM (Na naar 
eoMon aih anothar aacraB occur) «Mla your 
aacnB was on tha taOafoy? 

C. <OfttiaiaOet4]naarooMaionaw»arro(har 
Ofcrall hoa many occurrad/Did ttaa rtaar 
ooftMn «•! andhar aacrafl occur) ahia your 
aacraB waa on Ha ruTMoy? 


• rcMiaaouKH COUPON . .. . 

PAHOaTOOBM. 


>d«> www o»aWTiME* L L_l i 

TMI AMeUWr ai 0B14A CAMMOT n ORKATBa TMAN TM 
waouwT ai 0014 . 


• ONTAXMnv 111 ! 

fMK AMOUHT n 0B14A AMO 0a«4a COMOarBD CAMNOT 
at OMATBM TMM TW AIIOUMT ai Oi«4 


• OMMUNWAV I I I 

TMt Anouarr ai otiAA. ot««ai AND oiMe eoMHMto 
CANNOT at OaiATM TMAN TMI AMOUMT ai OttA 


00616. Durino Ha !■! 60 day*, hoa many limaa did you 
ai pananc a a c ct aio n Of naarccBiio n afth anything 
OHar Han an arUmal. a ground vahida. or anothar 
aacrat aMa on tha groMFid? 


•oncMonoiaoocuJDaN 

Ba,araaTOAm. 


A Miacaara Ha oOtactt you coOdada» or naartyocAdadadtt? SPECIFY 


ormoociCTtON: My nad QuaaBoni ara atom aircraft 


0AH1. OurvigHa Mat 60 dayt, hoar ffiarryimaadW (an 

aapianaia h aBecoHr)onaHicftyouaaraapdolorco- 
piol uaa coma of alfaaar^m fual aa dafHad by Ha 
6Alt? 


0AM2. IHoamanylmaadWyou^Aooaplan ATC 

c i aar a n ca Hoi Ha (aaptanoHalcepiot) could not 
comply ath bacBuaa of ti padormanea Imta? 


a Aocon OEMVUCK NOT cca^Y arm . 


OAK), (How many imaaHdyeuQLoaa eight of anoHar 
aircraB from aMcH tha pAd or ocpiol Nfoa trying to 
maintain viaual aaparaocn? 

A tOtfhalBAKSltimaayouaacraBiottaigNor 
anoihar arcraB. hoar many oocurrodDid kwing 
aighi dfanetiar areraB occur) la mar^nai WMial 
condtxma d a nOao or laaa? 


aioaEardHTOFAdCiu^T 

DAMP 10 AMI 


aaiiMWMHALviauAccoNomoHa. ... LJ-XJ 
TWt AMOUNT aiAlOA CANNOT BBOWtATBR THAW THt 
NDogwraiAMS 


OAMt pioa many Omaa did youIhuHrartanOy land athou a la#® vao CUA/WNCt 
daoranoa al an arpod Nfth an aeOva control lowar? 
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Gtof^ AiKaflon Sufvy 




OAM«. OwngtlwlMieOday<.n(wmwiylini«M(an i ! I I 

hit c opl w ) on laiWch you wm* a p<ol or cd- 
pM raPi^trtinp/ b«igM tHutctf «(9ioul A. T.C. 
dMrano* M an airpoil wtth an acttv* control 
(ewar? 


ATC • MR TRAfnC CONTROL 

OAHt. {Hoa many Mwaa did youf inatfuartarRy ilriwta from aotviAnONt > j . ’ 

an aaagnad routino or AT.C. vacttr tor ana n*HAa 
ormora'? 

ATC • AIR TRAmC CONTROL 

OAH7. iHowmanylirTmOtoyou|Taliaoff«ahanoul<«M atwctowcxn-oa-iJwrcaNrtR 

oamaro^praiity? oforasttv I i I 1 


OAHt. |Hoa many ttmaa^ you! Takooftcwara oig tr? •TM«wa.W«KXT I I I I 

HlUCOaTVI IMIF TO AM^ 

AiaaiAMi coNmuf. 

OAH»^ AMnANCONLV [Hew many Rnat <M you|^^ «WTfiaa>(K}PERGGNFKUUTiON LJ !_! 

Corwnao ca tofc»oR rd an Impropar aifcriR 
oonflpiinaon? 


OAN 1 Q. Aa a rammOar. 9>aaa ouaattona at! rator 10 Rw fliAijKiAt.ATtrrm I i I i 

laKOOdaya CAAir>Q<hataat600ayf.hcii« AMOUMtaiQaibAMit. 

many tanaa did <an oTptona'a haaooptar) on majoortir ooinaKii. 

atiich you wara a plot or oopM ai^artanoa an 
unintanOad unuauai aOluOa tor arty roaaon? 


UNURUAL ATTITVIOC • ARCRAFT OOTSlOe 
NORMAL RJQKT PARAMETERS FOR CUM8M0. 

OeSCENOMO OR TIKVMNO 

OAHim HEUCCIPT 1 R 0 M.T IHownwiytirnHdMya^ tvOWRPMWWBWO I I I I 

EKpmne**«iMlewrelorRPMiMnwig«orany MucomiitKirTaiuni. 

miofl? RPM « REVOtUTWWS PEW MHUTta eontwi*. 


0AH11-A. AWPLANC ONLY many ttmMdWyOl^ aSTMLWMWMaiVtlCXNVUfSNACTIVAIIOM LI I I 

Ej^artanca an unintentionil atal or vaU atal 
warning? 

0AH12. OwtogtholaatlDOaya. howmanyiimaadidtan aMKARCOtusocsoROCMO till 

a>rptono to haicoplar)onafRichyouwofaapflotcrc» raiAaiP iAaa awPTOAit^ 

pM nawly coMa wtthlarrain or grotf^ obatnicsion NauoopfiRnPTOAW. 

or artrat aMa airtoma? 

INCUIOES BLALOtNOS 

A- (Ot9)a|fAHi2] naoreeCRonawtt terrain. tATceROuaHTTOvouRArrcNtot I I I 

ground obairuesionerwiraa. hew many tmi amount maniia cannot at ontATVi than tmi 

waraAAtoa thw naor oetoaion wan tarmn. ground AMOUNTWAmi. 

obatruction or aArat>^roiigh( to your a ttan ti on by 
AT.a? 


ATC • AM TRAFFIC CONTROL 
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Q<ngfKAvti>oftSun»^ 


112 


B. <OfttM(iAHl2| ncarooMomwthtvTam. 
pfOund obdtmcUofi of ortfM. how mony 
nwofofWOa thM fiMT ooBoion ««h torrom. ground 
obolnjction or imtoo) doMoiwl itvoMgh diroct 
tlgMng of fho grotfid or obtfruesort? 


• OCTCCTtOT>4»CXK»*D««CT8iaHTMO LJ_LJ 

TMi MiOUMT M AMI lA MIO AMI S COHIMIO C4MN0T 
H ONSATV THAN TW AMOUNT « AMIt 


C. <OHho pi AH1281 noof colicniw hour mpry 
nwoKod JuM ««o^>id thto noor oobaton (rrvofro 
pjotoArw?) 


• nA^^iNavMies 


IMI AMOUNT n Am» CANNOT M OMATW THAN 
THi AMOUNT MANIl 


OAHISnA. AlfVWANC ONLY {How moriy Brnoo 40 you] •CNOOSMnHOCAAk^ _ I I I I 

iTMiOwwtNNtfy croM tho rumMoy throohoid <fcN<ng tho 
Mr^Ong ooofOOcTi am tho lortdiiTo goor up? 


A. <(Xlti#(iAK13]timMyeuBppreoehod«(lhtio 
iMiOng poor up. bow morvy wnooTTht ttmo you 
■pproNchod «Oh 010 twiOing poor up^ did you 
ictNiWy lONd Nth 9m gopr up? 


• UMDVNTHOtANUN LJ— LJ 

THI AMOUNT M ANfM CAAMOT BI ONKATMI THAN 
THB AMOUNT MANtl 


0AH14. OunngthoiMt90doyB.hONmooy«mMdld(on iiNCiBANEDAMarNC* 1 I I I 

■rpMT M M h tloopMrl on witNch you Noro a pdol or co» 
pdol InodvortiriPy ontor odipoco tho oNcroft hoa rtot 
cMorodtor? 


0AH1& How many bmao did you toM track of tho natural ALMaioaM I I I I 

hortBon duo (0 roduoad vIMbMCy tiMa tying urdor 

VlaiMini^RiJoo? 

WMi FUOHT RtXES ALSO R£FEfWEO TO AS V F Ra 


[ prmooucTTON: Tha naoi taw Qua ot io n o oro about oilHudt da Ha ttoao. ^ 

0 * 01 . Ho>inn|ftin«<k>1ngth*MaO(tayi<M(ln •M,tmj0i0tv»lIl0>«_ I I I I 

afpMnaM hoBooplar) on Nhcti you wart ■ pdol or 
COM bMdvartantty davMla tom an aMuda 
acalgnadOyAT C 7 


OAOL ASKOM.YPtASCIA10COIIAtCrA11C»aOTMm 
SKIP TO ATI. Earlar. you irdcMad you Aon 
< dAdC«AiKOfiAOC*Aii)l/R fiM (For hoN 
many ofthaao MgMaFor ONB Mgrt). dW you daaeand 
baiow MNUmum Safa Attuda Nhan you wara Nol 
folDNaig A.T.C. radar vaclart? 


• NOTfOXOMNOATCtAOUIVCCTOaS L ,1,1 J 

•MOULO NOT M » TMAM AdC *Atae ON APCMMIC 


I PlTTiooucnoM: Tha noad fpw qu oa h o n a araaOoulInMfa c MooawflBiirtraWccorUrot 


o*Ti. Ounngti*lM«><tt|».h(MtrnMytimM«M(in ,uHA«tiTOca«ui«»re«n>.ATC I I 1 I 

oapianMhaBeeplarlonNhchyouwaraapAotor pa^iHPTDATL 

eopdotiFiabialooormiunicaiaNthA.T.Cirialinia- 

crtbcal a b ia ac n bacauaa offrapuancy conga att o n ? 
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QowbI A v>on Smvy 


A (OftfHMliATIIenwyouwmuniMtlo 
oomniwiicaltaiVi ATC In « 
i><>cn b»efc»t»c*ir»Qu»ftey congttwvhow 
rmny oocurrtdM ttw thn* you war* unaM to 
oonwnunicalo wth A T.C in a tme ortcai 
■ajaa cn baouaa ct co npaa foo 

ooaa) wMla on Bha 9ro<*ndr7 

a (OfthaaattATi}amaay«u«waunaWaio 
oonwnunicali a<h ATC. In a tima<HDcal 
afejaaon bacauM of friQuoncy oongaaBDfv Now 
many oceunadCM tha tana you «ia*a unaM to 
oorranunicale afCN AT C in a lana crtcat 
aftiMion bacina c> Iraquargy oonpatBon 
oeoa) wMto airboma ki o»a lamiiMt araa*’ 

C cOftfiaoa[#AT1)timaayouwaratmablalo 
oorran unica lawthATC. inaiirna- cn b ea i 
mmann bacauaa cf baquanqf conpaaboa how 
many oeAaradOd tha tana you wa*a unabia lo 
oommunicala «th AT.C ai a lima cbctf 
atuason bacauaa cf kaquaney eongaalon 
oook) wMa on rooto? 


OATS, many ttnaa dd you| % at an undMirably Nigh 
aMuda or aaapaad on approach dua 10 an AT C 
ciaaranca? 

TMS MCIUOU iVT MAY HOT K UMrrtO TO *«UiM 

omar ApoHOACMfs 


OATA iHowmanytvnaaOdyei^iaawaaeommunieaiiora 
baouarwy wth AT.C to gia a aaaffiar bnoAig? 


OATA How many ttnaadunngthataoiaOdayawara you 
iniortoad ttiat (an arpianwb haieopaar) on whch you 
warp a papa or copM m baa d a tranam aa cn bom 


A (Of iha y AT4| amaa you mHaad a Irarwcnaacn 
Nom AT.C. how marry oecurradOil lha Uma you 
miaaad a tranamHaion from AT.(X ooeur) dua to 
batog on lha anong fiaquancy^ 

B (0>that»AT4jOmaayounaaaadalranarnwago 
bom AT C. Now marry oecurradrOai lha Bma you 
mHoad a banamiaaioo boan AT.C occur) dua to 
Mgh oocbpb ooiaa? 

1. (Of fra (d AT48] amaa you mmiad a 
banamHaion dua to eocbp* noHa. ior 
how many wara youAHOra you) waarbig a 
eonHniinieaiion h aadaal a( tito dma'^ 
THIS INCUiOES ICIMETS MITH arTECAAl 
HCADSCT 8f>EAICRS 



TMe OQMOBWO TOTALS SI AT1A AW ATTS CAMaOT 08 
OMATCa THAN TMt AJIOUirr H An. 


atWaLEBINCIUTE 1111 

TMt eowaatto totals si ati a Ana aw atk 
CANNOT at OatATU TMAN TMt AMOUWr M An. 


at«»4ia.TnieiOMMmn . - — 1111 
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OATS How many vrrm <0 yow put d 6t$». 

inaocuralo or no (ntenraftan itoul r«t*v«n( NOTMto 
(MMrno>7 

NOTAM8 ■ MOTICeS TO MRMPl 
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QfwH Av^taton Survty 


1 


MTHOOUCnON: ThM McUon of th« Mtnrtaw to Dull on nvMtfwr-fvMsd mum. bcginntng wMth«f ptomlng 

for your IIIgMo. JuM m • rwnindor. wi oro ill o«4y Mfcing about ovona tMI you fxpanrteod 
OunngihoiMt 60 bvyi lying unMr FAR (Part 136/P«t It Part136andPart61)M<inavplana/8 
hilBOOlif) plot Of ccpiot A^mv w um tfia tarmi tight* ttirougbom 6i»o wlaoiiaaf to m aa n t»a 
portod of Bfna boMoan oaoh talaoit and Mino. awan « OM tlgM OTM b artert aueh M for 
rabucton taatfifig atudanli to tond or tfidattolinQ *toucli and ooaa.* 


OC1 Earlar In tha Marvtoar. you Mcatad you mada 
IM6« A10 airpton«A»* A1 2 Ni c odibt takaofKa) 
dtfing tha lato 00 days. (For fioar many of th e ai 
Aghia (id you obtain pra^hpm aoatbar Monnatton? 
/On tM light, dto you obtain pro b gWoaalhar 
Informatton?) 


• ruoKrtMCATManaMCTMO LJ — 1— 

CANHOT at OatATM THAN Aa»A1l you AMFlANi 01 
At* All fOa HtUCOaTEft 

PA aiuaToca 


A. How many tanM wm pralight mformalon obt a mad by (READ QUESTIONS)? 


1 Comwarcial TV. radto. or cabla wffiar 

broadc aat mat waa not aoacAc to m ta h en 

2 . Comma mi al TV. radio, or cabla waaihar 

broadcaat (hat WM toacAe to a«toben .......... 

1 Company pfowdad waathar from a 

dapaichar 

4 OUATS(OOwbati)Of otMCcmputar- 

accaaaad Mtobon waathar aannoM 
(OUATt « COMPUTCR-8A9ED VUEATICR 
SERVICE PROVnSO 0Y THE FAA) 

5 ProMOordad FbgM Sorvtoa SMton Waadiar 

FUOHT SERVICE STATION ■ P S S 

6. VarbaibnannoawlhFAAfhgNMrvloa 
Mbon apadaiata 

njOHT SERVICE STATION ALSO REFCIVtCD 
TOASFSS 

7. MyouobtatopraAohlwaatoarlnlDrmatton 
in aoma edwr way? 


1 1 I I 
Mil 
MM 

MM 

MM 

MM 


- 






M 

(BQPTDCto 

a 


a. How did you obtan tw watohor Infocmodon? SPECIFY 


SPfiOFY: 
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G^nyil Av^HW Survfy 


OCt. rONLYOMCQUC8T10NANtVI«MOMC1AV7.SIGP 
TO C2A. You taid you uMd th* tatonwriQ pro fl^ 
w ottfiar i nte rmi U oft ootxcoo ip »o lot 60 doy> (UST 

rrcMS cooeo one ok mohek m ciat-t). vkocn did 

you UM moot rooon^ 

COOC ONLY ONE 


A. Hour undomortdobio woo tho ooothor 
Wbmotion you rocoNod moot roooftOy from 
<SOUfrCe USTtO IN C21SMGLE SOUKCC IN 
ClAi'T)? you oty • iwoo (READ 
OPttON 8 )T 


B How occufolo wo thi woolfior tmown tt oH you 
rocolvod moot rooondy from (tOUKCf USTED M 
OMOU 80UKCE M C1A1-7) In roloOoo to tho 
woofriof condfciono you orKOunOorod Oumg 
•qM? WcMid you oay 80 Mormotiort woo (REM) 
OPTIONS)? 

C. How mucK limo ilip n j O otw to n your moil 
mcordwoithorono^ondftofrmtoltifcooff? 


OCS. fri wtacO itMo or otatoo Oo you p io u aily fty? 

RSCORO UP TO 3 STATU USMO tTATC OOOC U«T 
KLOYV. # PACT OrVU OTHER TVPC OP AMSMCR 
(E.a. HOKTHEASr). RECORD 

I. RECORD OT>et ANSWER 


OCA Ai a mmmdor. wo aro oUl only aoUng ibeul ovont 
that you ai 9 ortoncad flymo undor FAR (Part ISSiParl 
•liPart 135 and Part 9i) ao (ar> airplane hoiooploO 
paol or eopM (Of tha ISAS^AIO arplanaiAS^Al 1 
haAooplar] tafcaoRi you mada during tha loot SO daytv 
how many of thoM fUghii waraAUio tha tafcaoff you 
modi dw^ tha loot SO daya) oonductod uraUr V F.R 
(IghlnAio? 

WR^VtSUALFUOKT RULES VISMJTY GREATER 
THAN 3 IU 8 AND CEAINO GREATER THAN 1 m 
FUT ABOVE GROUND LEVEL 


oca. Oo you. or your organmnon. apply pra-ngNV.Fil 
waatharmininvnatfiRammoraconaarvatwathan 
thcaa raourad fry tha F AA? 

IF PACT menioNs ifr »cre. let hmher knm 

MLL BE OETTINO TO IFR LATER IN THE MTERVCW 


PagaZ 


CQ aa gna iiLSOLRoeawoT arBCn cTOAviAtiON 01 
COMUBKtMaOLRCCSVeCinCTOAWLTIOM. 01 

COaWANV P HWIOBDWEATMgt INFORMATION 01 

OUATB Oft OT>€R COMPUTER ACCSS8G0V«AT>«I 0* 
TIM PNE4tECOR08DFUGHf8EmnC6 STATION— - M 

A VERBAL FAX ■tWMO 01 

OT>€RFf«ruOMTa»€RIAMWN ^9f 

m . ... ar 

DK ai 


HM M at urdimndiwi — 



aTANEOrFauVOCA VTR I ! i I 

CANNOT aC ORtATBR THAN ANAM fOa 
AMPLAMI OR At • All FOR NtUeOPTta. 


NO 

IMPTOOR 

a 


fWcwTnoR 


OK 

MOP TO OR 

a 
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Q^ntral Ai^iBon StfVif 

A. Uwdtr tho— mcr» wwff u r 

mMrvMTV. VtM it th« numbtr or mHM 

of ¥WWniir yiw or yoi# oramMMjn i^quro? 
r«Pf PTTVV mJMCA. NCUAPIO OCCMM. 

B Undor thOM mar* oonoorvolMi WMthor 

mMmumt. it tho mirirTvn eoMIng In foot 
thit you or yov orgar«aOon roqurt? 



OCe. AQtin.youindlcalodyDumoqt|IA6MlO«irplv>^ 
A»M11 hoieopttrl ttMofl(t) M t ptot e« copiot 
durviQ tw ImI BO dtyt. (On how marry of lhaaa 
(lgMtA)n Vial (IgM} did poor waotiar latuih you 
bang vtcfc d your poaiion? 


A. (For tit moat racaraBB**^ortM(lon).wfMl 
waathaMAyinnAta^ 

Tvat mTNR NtMCA. MCiuoaio otcaM 


OC?. ttihoaf marTyoflhatPAB* A 10aadtnadkB*A11 
iiaicopiai) tiQPit dM youoipartanca tpaliai 
dbenaotaoon from poor viaitrBly dua 10 oMtiar^ 
that fllQfd iM you axpanaoea apabal dborbniaoon 
from poor lABidy duo lo abathar?) 


A. (Hoar marry of that# figMa oocurrad at nigA? / 
CM that OgM oecur 01 flight? 


B. For (Via moat raoarMhaQ bma you axpananotd 
tpatial dooriantotori duo lo waatiar, aVM afpt 
tha aaomabd v w ipacy in rnPaa? 

TVn BfTttf NUIAEIt MCUfOMO OeCtUtt 


OCd. Again.youlndicatadyoumodatBAB»A10aapiand 
Afr»All halcoplor1uMofll(t)atapletorccpde( 
di^vig tha patt 00 dayt- (On hoar many of lha riaaa 
fkgMdOn that light) did you inadrtflarBy ordar 
inttumanl mataorologicai condiuont. or I M C., artitt 
on (arthaQ V.F R tigfv? 

IMC • INSTHUMEKT METEOROLOGICAL CONOmONS 
^SlBILfTY LESS THMf ) MttU ANCWOR CLOUD 
CEIUMO l£$S THAN 1.000 FEET ABOVE OROUNO 
LEVEL 

WR ■ WSUAL FUOHT RULES VtStBUTY GREATER 
THMf 3 ULES AMO CQUNO GREATER THAN 1.000 
FST ABOVE GRO1M0 LEVa. 

A (Hoar many Qmat did tmEM thia) ooor at 
Mghc? 


3 




atMOWIRTBMTaiC 


CAMWOT at oatATta nua At»Aia roa Awauait oa 
At*Aii FoaHtueoana. 
VAtiaaTOca. 
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Gfxrii Antioft Strwy 


B. HpwdU you most raeartAhaQ 

tnadvartant I.M.C problam? Od you (RCAO 
AMSMCKS)? 
cooe AU. THAT Amv 


1. HowddyouroaotvathallliC.protiim? 
BPCCIPY . 


OCf <Onhowm»iyaritta|«AB«A10airpMA»*A11 

hrtooplir] fhgydi/Oo 9m * 0 ^) you madt (ha 

IMBO da^. did aaathar c ondMcoa raaul m you 
conducarg a oo^aroimd or maaad approach on 
Ivdng? 

A. (How many omaa aaa (ha 90 «ound or miaaad 
ipproact^Aiaa this goaround or mtaaad 
approach) dua (o poor vi ilb t y? 

PftOMPT PHOT CCNOUCTf 0 OOAROUNO OP 
IBSSCO APPROACH ON LANOPiO DUE TO 
RCATHCR OOttOmONft 

B. (Howmanyar)aaa(Badili 90 >aroundormaaad 
ipproachA^^M this go arn^d or (rtaaad 
approach dua lo Mgti wte>da? 

PROMPT PhOTCONOUCTEOOOAROUNDOR 
IMSeO APPROACH ON LANOmO DUE TO HIGH 


OCto {Onhoiiirmanyo(0ta|BAB«Al0aapMrta(AP*A11 
fratoopM tiM RghO dd pcraarung 

aaadwr conddona roaUth you dtvamno to on 
•PamaiNa landyig Ma? 


A. lOittia moat raoardA^nihoQlkgmarhari you 
dhartod to an aRamaPvo laridmg aAa. hoar dU 
you dMarmina that tha araathar a«a woraantng? 
Old you (READ ANSWERS)? 

CODE AU THAT APPLY 


1. Hourddyoudatarmnathawaaiharaw 
noraaning? 


8P€C^ 
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THE rOLLOttmO QUCtnONS AHE FOR VFR RATS) PILOTS ONLY 

OCTriaANior«toMOursncNAi*M3r9i srptocis 


0C1« l<rn«q u »gto n <«r»iNCUIn«N w i»weSyliis fiw 
gang to aNta Itw q u MNarw about imirtfTW« ftirinj 
now m^r oonSwelaS on th« |AAS«A10 Mrpiina^ 
ASMtl hilPopNrl Ifcgfti you >— » — (m mtpafmf 
NAeeplNfpAoloreepMovortNlMlMdoyL (On 
hew flionir of t« 0 M SgM* ^ M Agho. M you And 
%wig V F R OKO* 0 eleutf tfocA. oomolmoo 
eoAM V F R on lop.* wNm you riad a» poneMs tfw 
dsud dod( n ofSorio Nntf? 

VfR-VI$UAiru(R#TmAJS VISWOTVOREATIR 
1HAN I UUU AlC CeUNO OACATBR THIM 1 .000 
r»T AROVE ONOmO i£VtL 


•vmoNtap 


1111 


CAWIOT m —ATI TNAM AAMktO POA M 0 mjm OR 
A»»Att POR HiueorTvt 
PCMprocti. 


A. (Oft tfw AMI roeorsOn M) RgM Rtton you 
•or V F R owor • doud Oodk how ON you 001 
•wo^ ffio ooud aodi 10 lintfT Old you (REAO 
CATEOOMEST? 

COOCMLTlMrAPPVY 


MltVATC IWN*RWUIOldNR | W>OW H R<Ql 
AehNrAT.C mN and dodNUd en wwwpeney 
OweandXduBighdwaeudiwRiouieofiMipg 



1. How dd you 0 N Owougfi tfid OOud dock 10 

Nntfr 

SPtCW 


OC 1 L How d«ny hour* of wdOwnont tradwig hdMO you > KW J^oriaiiuiWiwT iRUiwul — I I 1 I 

roooHod Nneo you bdoon to iyT 


OCM. HM»w<)rhoune<nMng»M|rOul«(«M«M •KM>tOHCfU«l.««11MWnill»«9 I I I I 

■ctudi I tl C conddcni Nnod you Odgon to Oy? 

■c • MiTmacNT MrrccROtooiCAL coNomoNt 
MMUTV LESS THAN S MkES ANOm CtOUO 
CSLMO LESS IHWt 1.000 FEET AROVE OROIM} 

IfVB, 


OCtL 


How Nng ijo woo your Rdf Wd tr umonl SMUng 

rmieauoes MEMM. aiQHr Sinews 


JJ- 


.LLL 


.LLL 


T>C POU.OimMO QUESTIONS ARE FOR S1« RATED mors ONLY 

ocTimt»csntoNoic8Tx>hiATA>i cmcRs.aKrrool 


■fTROOUCnONc MynwdQMWiOond<rodOoi<lndlndmwtftrtn9.NowHm90iStOdafcd<OwduMBOnd«60ul 
um m mfdSymgyouwdyhww cooduefddXon t w p id n U P ilcog Hr i pSoiofeopSat 


OC«L (Onlowmanytf Sw|MS*A10NiplMA»*A11 

hdloepf^ SiF^On ft SgfO you eondudod w Bid 
M SO dM dri you Ra •! I F.R IhBht pNn? 

riN nstMmnt ruOMT Rujs 


mPueMTPUMS ^ I ^ ^ 

CARWOT at Od t AT tW TMAM AA*AN PQR AMPlAAR OR 
AA*At1 PCR NdJOOPTRk 


WCNORTOeid 
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A. (Of iri«M |f CIS) flghift whan you fliod art IJM 
•gN plon. how mony had I.M C eo nd fl cnt ai 
iMat port of Iho timortWion you Mod tfao I^.R 
MgM pMn. M t htvo I.U C co nd li o oa af loooi 
port of iho tano? 

MC • M5TRUMEMT METEOROLOOCM. 
CONOITIONS V«S«UTY LESS T>4AN 9 UOfS 
AMVOrt CLOUD CCfUNO LESS THAM 1 DOO EEET 
ABOVE OAOUNO LEVEL 


DC1E Oo you. or your organoaPon. apply proMght U Jl 
wo Mh or ffrtumiaoa thot an moro co waory u lvo Pian 
Biol foqmrod by Pio F AA? 

FR« WSTRUUEKT FUQHT fOAES 

A UndortttooomorooonoorvaOvolFRwoolhor 
mowmana. otiot la tho mMman riumbar of 
loilooolv f aM ty youoryotaofpon ttafi on 
(oquvo? 

B Unrtor piooo conaon^dvo I.F.R woobioc 

nvna mg m. whol M Bta nUnlnHim eoUng In foot 
youroquao? 


OC17. F CIS IS A 74 OR A SIV TO CIS During tho loot Mghl 
you Sow wtioro you Mod I.F.R. did Iho oircraA havo 
(READ QUESTIONS)? 

A WooBior rodar or Siundonlorm doCoctton 

aouRmonc — 

B AiOnpOni wOaangiMng lwmiif 

C. AIRPLANES ONLY. Arai^caiooqu^riiantttiaila 
approvod for flghl in idngoortddiono 


OC1A (On how many Of Bia Id FLIGHTS M CIS) Ughtaohon 
you Mad on 1 F R MgM pMrVWhon you Mad tho IF R 
MgN plan). dM you By an mabunont approach) to land 
m I M.C ? 

BK • aiSTRUMENT METEOROLOCNCAL CONOmONS 
^SMAITY LESS THAN 9 ML£S AMVOR CLOUD 
CEABIO LESS THM1 1.000 FEET AMOVE GROUND 


A During Bio (Mat) flght whom you Saw an 
InoMananl appmoch to londBig in ISI.C 
oonMDono, What typo approach wot Sown? 
DO NOT READ UNLESS REQUf STIO 


1. WhN othof approoc h woo Sown? 


SRfClFV 


0 


MconoTO. LI 1 J 

cAMaoTos ontATM nun cit. 



LLA<aCY1BjlBrTLAN00«»VrSfBR 01 

VOR (VERY-HlOMfREOtENCY OtaoiUOlQQ OB 

RNAV (RAM MAVIQATION •ILnart Ot 

OFA tOeOOEMC POSmON SVSTBB 0« 

LDAAiBtaOCaCSrTALTmiOQ 01 

EOF. lOaFlFMD O WE CTIOHA L PiCUtn 01 

MOAprOHOmeCTKMN.aCACOIO 07 

BACKCOURSCILS _ Q» 

SOMEnONGElSE ptMCCtVtt} 00 

— - - - - ^ 
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OtrnfilAi¥Htton>urv»y 


P19»7 


Ounng tw (Int) ftoM wftM you fl*w 
Inttrvncm •0pre«eM» lendi^ in I MC 
oondfions. wm tw caing. M IM. <}unng 
tha appfoaci)? 

Durtno **• W) wtMT* you Amt on 
molnjnont opprooc h Id londng In t lA C . vAtai 
woo vw vWMty OwlriQ iho oppfOocMn mioo or 
Rvm 

RVR •RUNMAY VWUAL RAMOC (W FCfT) 


iCCUNOOVtnuUCNTLAMWOFKT ^ ^ ^ 

m 


•wauTVMnmiMNTiaut Mil 



« m 


OCit. SWF TO 01 . You tn^clo tf that • •«TnjM«NT Min 01 lilt 

you modi i« C 1 8 ] flgM(o) on wtieh you eondudod 

on molrumont o pprooc ft lo londinQ in I.IAC diRmo tho ir ■, mo TOOt. 

loilWdOyt (HowmonyoftMoapprooclM 
mtnMm Mi opprooc l i) oondudod hrtOor FAR pan 
»1? 

OCM Ai you may know. Via f AJi cunontfy alowi pAota tying undar FAR Part to conduct matf u rnanl apptooctiai. 
bd not landingi, afian tho waolhar condWona at ifa inalnjmant approach landing facdty ara baton landnQ 
mtnimuma. 


A. Afo you anam ofthaaa ragulatlona? 


B. You>atMGalad9ialyouffladaOtCti4 
inainjnanl appmochiotl in I.M C and undar FAR 
Pan t1 during tha tad 00 dtya (HoamonyoT 
ihoao Mai dU you fly VtoDid you Oy that) 
approach atti Via rapo rt ad aaathar oonditiona 
batow dia minMuma for dial approach aa 
atoaad by Via F AA7 

C. (On lha moat raoarVOn that) approach did tha 
airpon haua on-aaa waaViar raportngf? 



9 : 

m. 


.1 

.7 


0 (OtfinghMrnanyoIVKMapproachaarOwng ia«Tm»iT moriMuac l_LLJ 

Via a pproac h ) waa tha aaatha r abova fha 
Mnanuma whan you tandad^ 
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Awten Sufvy 


Pag»i 


SCCnON O: QUCSTlOMNAMtE FEEDBACK 


iMTiloOUCTlolt; I oftfy fw ■ coup>» mof qu—ttont f>d th— an about your r— rtont to ffi> tuvy w juK 

dort« 


OOt. How oonAdart «« you Biflt you aocuraMf oounM dl 
of iho Mftry-fvltM M I Micod you «bour> 
MouM you toy you vvoro (READ OUESDOHSy? 



002. VFtro ony of 9m ouooaono I oikod conAsino. poorly 
worOod, or omMguout? 


.(•VTO09. 

.<WTOO<|. 

IWRtOOO. 


.1 

.0 

.f 

4 


A. Couid you ptMOO dMcrtw theot quooOon 

problm7AfCOAOVCRBAmi ATCOURlEDON 
OF OOEIWIEW. ENTER OUBSTION NUMBER 


001. AiodwioanyullityproMomohappontnooilMnBM 
nooonal o niOnn tytum tNi I did rm — h obou bm 
(hot you 0iM( rnoy bo wortp) ooking BKMt In luturo 
•urvoyo? 

A Whit in thoM problomi? 






BMP TO 00 

f 

OR 

IMTOS 

i 


sPECrv 


OOA OoyouusoOiOlnlimiiBhamo? 



OOA Do you MW my othor commoro or t uQQo m oftt ibout thO o^voy? RtCORO vERBATnt 
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G^rmwt Avirtion Sur^ 


OWVfT Again. Oiank you vory much tor your ttnw Mid your hulp wNti tfiio ourvoy. Your Input wil twlp thu 

a v tu tton mduoiry a graut dual to m aaaura lha (aval of aafaty In itia aviattoo ayatam and uHlI ba liald In 
eonfldanca. 


QUESTIONNAIRe LeaOTH: 


QucsnoNNWRc i0«OT>i pMunn 
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•MIT90«XM.T1M 




\JS. K0U6£ or nCPnESEWTATTVES 

COMMITTEE ON SCIENCE AND TECHNOLOGY 

BtAlf 21» MYMRN HOME ones MKIMQ 
WASMNOTO^. DC SOtlS-CIPt 
lXSlZlt>«|71 
TTy:BB»m-M10 


October 19. 2007 


Dr. Micbod Oriffin 
Adnunistiittir 

Nstioail AercHtfobes nd Spooe Administntion 
300BSt.N.W. 

WsslHiiftoi^ D.C 20007 

rv*r AAii lut M r atnr 

More foor yetrt sfo. Ote Nstiooel Aeronwtke aod Spooe Admmittntioo (NASA) 
wUb BaneUe Moaociil Imbtuie to ooadoci surveys oTt,000 evietkn pOocs 
ooooaninctsfoyevtiitiOiei they had expcticooed. Tbeparpoeeoftbe surveys was to 
**provMk rdiable safety data for unp rov iq g aviatiaii salsty.*' The pAots were that 

ilMif partictpa&oo would Tattber improve aafoty for yvo. your ooUeaguea, aod iIk 
mviwtiart mkXtiT (Uodaiad ‘‘Inuoducsory Lectsr** to paiticipaats from Mary M. Coodois 
aad Linda J. CoosteU. pre^ co*aaoagcn). 

Hw survey appears to have beeo part of the ooeb larger Aviatioo System Mocntortiig and 
Moddhi^ Project, which was to apbd pa te threats Id safoty and mana^ riik m the 
aviabcQ world. The firal step in the prajeol was to modtor the system oootisiaoaily aod 
coUeei, codify, and daaniy safety bddenc data into r epo ai tories that can fhoi be 
analyzed for tmo aviaboii aafoiy. 

One of (he monttorins toob was the Nadotul Aviatioo System Oper a botial Mooitocint 
Servloo (KAOMS), a coettprahcBfive and coherent survey of the operaton of the B%iacioo 
system (ie.. its pii^ oootrotleis, roerhanics. dtspatebm, fll|hi attendants, and others) 
ootreftilarba^ According to a descnpcjonwricteo by taro employees of NASA's 
Ames Research Center, There is proven valoe in viewing the aviatioo system through 
tbceytecfttsopenfiocs. NAOMS it a longitudinal survey that will track safety treads, 
momtor the impact of technolofical and procodtarel chan^ to the NAS [oteunal 
aviaitioo system), and oaatnbote to the devdopmeot of a daia-drtvco baiia for mfoty."* 

NASA spent nuDions of doUars over three yean to oocrtact 24,000 piToia seeking 
r espon ie i to foe survey. Yoor reaesrAars told Subcocnmlttee staff that foe rcsponac rate 
was approximately 80 percent NASA appareody canceOed the survey many yean dtorl 


* Irving C Start v aod David A Mahi( "Aviation System Moottonng and Modeling 

Project,*' 200S«01*2975. It appears that surveys ofothadrliDe employees were not arete 
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of oompletioii, however, witboul emn p flin g or analyzing the resulta. NASA also 
apparently cancelled plans to conduct similar surveys of ground crews, attendants, 
controller and others coDoeniing their expencoce on safety issues. NASA’s stated 
reason for canceling the surveys and not analyzing the data already coUected was that 
NASA did not have the necessary funds. Aviatian safety should not be a luxury for 
NASA pursued only when fends are abundant; one of NASA’s primary miasioiis as 
established by statute is to improve the ’’safety snd efBdency of aeronautical . . . 
vehicles” (Sec. 102(dX2) of the Natioaal Aeronautics and Space Act). The data iqtpears 
to have great value to aviation safety, but not on a shelf at NASA, ap pa rently unr^ 
since December, 2004 

When another party requested the survey material under the Freedom of Infoanadon Act 
(FOIA), NASA demed the request, claiming that it wu “commercial” information, and 
that “release of the requested data, which are sensitive and safety-reUted, could 
materially affect the public confidence in, and the commercial welfare o(i the air 
carriers and general aviation companies whose pfiots participated in the s orvey” 
(Letter dated Sept S, 2007 fiom Thomas S. Luedtke to Adam J. Rapport; emphasis 
axlded). That staled reason does not appear to fell within any of the exceptions under 
FOIA to the requirement to release requested infonnatioa. 

Your attorneys told Subcommittee staff that there were a number of other reasons for not 
releasing the raw data, indodingccofideDtiatity promised to the surveyed pilots. That 
stated reason is still less persuasive. All personal identifiers have been stripped fiom the 
data, arkd, in their communication, Ms. Connors and Ms. Connell promised only that a 
pilot's answos would “never be connected” to his or her name (Undated letter, stqjra). 
Additionally, a smaller sdf-iepoitcd aviation incident database is described in NASA’s 
own web site as a ’public repostoiy.'* 

In addition to being outside of the recognized exception to FOIA, the reasons NASA has 
given for not releasing the data appears conliaty to NASA's missiQa. The “safety and 
efficiency of aeronautica]. . .vehiclea” is part ofNASA’t mission; piotectiiig airiines fiom 
public concern about safety is not If NASA has infemnation about questfonable safety 
practices of aiiiines, airpotts, the Federal Aviafion Admimstration, pilots or anyone else, 
you should hsve analyz^ h promptly and made appropriate recommendations, or you 
should release the infocmatiem so the public can ni^e their own judgment about aviation 
safety. 

Now, almost three years after the last survey was conducted, NASA researchers say they 
want to analyze fire data and release their findings in a iqrort after alL The new source of 
funds fin the analysis is not NASA’s and staff now voices a strong desire to prqrare a 
draft report by the end of December. NASA would then review the report, e^«nally if 
the report includes leoommendstions for improving flight safety in commercial aviafion. 
The process of review will undoubtedly add many mote monlhs to the process of 
ddivering a product to the public. 


• “Aviafion Safety Rqxnting System (ASRS), http:y/human- 
fectors Jucjiasa.gov/awardsjpubs/facttheet_view.php7fimtsbeet_id-37 
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Moreover, NASA’s staled reasons for not releasing the infonnadon suggests NASA 
places a hi^ier priority on the commercial interests of the aviadoo industry tharr on 
pubhc safi!^, which gives rise to questions about how NASA will analyze and present 
the data. 

To help die Subcommittee understand more clearly what information NASA collected in 
the three years that it surveyed pikas in the NAOMS pnqcct, I hereby request that you 
provide the Subconimiltee with a copy of the questionnaire that was used for die pUots as 
pert of thcBattelle survey. If there is more dim one itaadon of that survey, please 
provide a sample of each. Pleaae provide that documenlfs) to the Subcommittee offices 
in B-374 Rayburn House Office Building by S pm. on Tuesday, October 23, 2007. 

Please also provide any copies of briefings or presentadon materials that the staff at 
Ames gave to the Airline Nets Association or other consdtuent members of the 
Commercial Aviadon Safety Teem (CAST) since January 1, 2003. These materials 
should be delivered to the Subcommittee offices in B-374 Rayburn House Office 
BuUding by 3 p.ni. on Tuesday, November 6, 2007, 

Finally, please provide a written explanation of the budget decision to terminate support 
for the NAOMS project All materials on this m at t e r reviewed by the Subcommittee 
suggest that NAOMS was a very woitbwhile initiadve that hdd the proinise of a more 
oooqitdiensive qiproadi to assfssing emerging safety issues for the flying public than 
anything else we have in place. Explain what factors led to canceUng die support of 
NAOMS and where the funds that would have gone to NAOMS went instead. Please 
provide that written response to the Subcommittee by Tuesday, November 6, 2007. 

I expect that we will ask for more information ooce we have received those documents. 

If your staff has any questions or need additional mformadon, please contact Dan 
Pearson, Subcommittee staff director, at (202) 223-4494, or ^th HoUeman, 
Subcon:^ttee counsel, at (202) 225-8439. 

Your prompt attention to this matter is greatly appreciated. 

Sincerdy, 

BRAD MILLER 
Cbaimian, 

Subcommittee on Investigations and 
Oversi^t 
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Ha«>quart»rs 
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OUA 

OLA/2007-«0*82:KSjmb 


The Honorable Brad Mills 
Cbaicmao 

SubcommitUe on lovestigaiion 
and OvssighI 

Committee on Science and Technology 
U.S. House of Representatives 
Washington. DC 20SIS 


Dear Mr. Chaitman: 

This is to acknowledge receipt of your letts of Oclobs 19. 2007, regarding NASA's 
respooK to a FOIA request cooceming the Nthocial Aviation Opentions Moniloring 
Service (NAOMS). In the lens, you request a copy of the questionnaire used for pilots 
as part of the NAOMS survey, copies of briefings or pteseotatian mat e r i als, and a written 
explanation of tbe budgs decision to tenninatc NASA's support for the NAOMS project 

As requested in your letter, we ate transmuting herewith questionnaiies in the NAOMS 
air carrier pilot survey. There ate two sets of questionnaires, one for oommercial pilots, 
consisting of four parts, and one for general aviation pilots, consisting of five parts 

We will endeavor to transmit die balance of the materials requested in your October 19 
letter by October 29, 2007, as you have requested. 


Sincerely, 




ID 

It Adminislraloc 

gislative and Imergovemmenlal AfTaira 


2 Enclosures 
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Oct 22. 2007 

SUBJECT; National Aviation Operational Monitoring Service (NAOMS) 


• What NAOMS Waa 

The National Aviation Operational Monitoring Service (NAOMS) was a NASA- 
funded research effort to derrMostrate reliable technology to provide safety 
decision-ntakers with tools to make more informed decisions. The corKept 
development of what would become NAOMS began in 199S-20(X). In Apnl 
2001 through December 2004, data were collected of airline pilots to support 
the NAOMS tool evaluation. In 2005, the documentation of NAOMS, as well 
as the transition of the NAOMS to decision-makers in the industry was begun, 
with both the ComrrMrcial Aviation Safety Team (CAST) and the Air Line 
Pilots' Association (ALPA). The NAOMS technical report took longer than 
antidpated, and is expect^ to be complete by the end of this year. 


• TImelirw and Budget 

NASA's Aerospace Technology Enterprise funded the NOAMS project from 
FY98 to FY06 (see below) The last two years funding vras provided to 
complete the project to transition the research methodologies to the Air Line 
Pilots Association (ALPA) and the Commercial Aviation Safety Team (CAST). 
NASA has completed the transition and is now writing the technical report 
evaluating the merits of the methodology. 

NAOMS FUNDING LEVELS 


F'trSB 

S0.5M* 

Fr03 

$1.8M 

FTM 

S0.5M* 

Fr04 

$1.2M 

Froo 

$08M 

FY05 

$0.5M 

FY01 

$1.1M 

FTOe 

S0.6M 

FY02 

$1.5M 




* Appminune. Actual oumben unavailable- 


• Reasons NASA ended the NAOMS program 

NAOMS completed its research objectives and reached its planned 
conclusion. The project's research methodologies have been transitioned to 
the Air Line Pitots Association (ALPA) and the Commercial Aviation Safety 
Team (CAST), and NASA is now writing the project's ending documentation. 
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• No data has been destroyed. 

Master copies of survey results data retained by Battelle in Mountain View, 
California, and copies at NASA Ames Research Center. Battelle has 
provided the (dllowing statement 


Aft- 

The purpose of this email is to affirm, at NASA request, that NASA has 
never directed Battetle to destroy the master copies of NAOMS survey 
results data nor has Battelle taken such action. Master copies of all 
NAOMS survey results are msuntained by Battelle in Mountain View, 
CA on CDs and other backup media. Copies of the CDs have also 
been conveyed to NASA Ames. 

NASA has directed Battelle to recover, or ensure the secure 
desIfiKtion, of any secondary copies of the NAOMS data that might be 
held at locations outside of Mountain View. This includes any copies 
held by present or past Battelle NAOMS subcontractors. The purpose 
of this latter action is to ensure that NAOMS conforms to NASA data 
security requirements. The essential goal is to bring all NAOMS data 
to a single, secure location managed by NASA. Battelle is in the 
process of taking this action now as part of the ASMM contract phase’ 
out process. (NAOMS project work has been accomplished under the 
ASMM contract) 

-Loren Rosenthal 

Battelle ASMM Program Manager 


• NASA's plans for the Survey Data 

NASA collected data to support and substantiate the demonstration of the 
survey methodology. NASA never inter>ded to disseminate the actual data. A 
final technical report evaluating the merits of the methodology is currently 
being written and is scheduled to be completed by year's end. 

NASA has transitioned the NAOMS research methodology to the Air Line 
Pilots Association (ALPA) and the Commercial Aviation Safety Team (CAST). 
NASA is now writing the technical report evaluating the merits of the 
methodology. The report will be made public when it is finalized. 

• This was a NASA-funded project and there was no funding from any 
other agency or organization in addition to NASA. 
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IW^IlN«U.mA» 


a& HOUK OF WPRfSENTATIVES 

COMMITTEE ON SCIENCE AND TECHNOLX>GY 


Sim »0 fUYlURN »«OUK <y«Kf MAAMO 
WAMNOTOK. DC iei»4K1 
D0812»«m 
HYimilt-MM 


October 22, 2007 


Dr. Midud Qnlfis 
AAnintitruof 

Ntfiouil A« 0 MUtks Mul Spoco AAiiinirtitioo 
300 ESI, H.W. 

WMfaiaitoa, D.C 20007 

Dmt Admmtraior Onffis. 

Rcoaetly. tbc had Iwpcfecd ao mveMgatwo into avutioii safioty pcognnu at 

Ow NdMMl A«Doant>ct and Space Adtntmamtioa (NASA). EartyUatwo^ 
r niHiBi Ba g iiiveatigative aiiff bid a tckpbooe coovanatioo witfa NASA ttafT coDoenong 
a aorvey of atrttoe piloti about aatey incidcntt conducted uoder tba National Avittica 
SyMem Opcradooal Monitonns Service (NAOM5). The Cccnmiaae felktwed op with a 
laoar requeatiag certain doogmeott and other infiaimatioti Tbeceferei, we were totpriiad 
lo read is the madia today that, after diat pbooe conftreooe, NASA offidala had diracted 
tba lead cootracaor m Amaa Reaaarch Ceoler for the NAOMS aon^ to archive all its 
oo Ifaia profaet, return ibe archived material to NASA «ul Ibeo purge it fooo 
tbeir ootspulen and files rMASA Site oo Ah Safiecy Survey * Aisodat^d Prtat, OoL 22, 
2007> 

By this letter, we are dhectiag NASA to hah any deetruclioo of reeocdi relatoig to the 
NAOMS proieci, wbefoer in foe posaeaaioo of egeocy or its cootmetors, and as 
de&wd in foe atfiicbed ^peodix. DestiactsooofdocumcBisreqwsiedMpstofa 
COegnesiotMiisqisryisaviolitioQofctiniinelfoderallaw.lSU^.C. 1505. 

As I am sore yoo know, ttds is sot tbs first time this year foal we have written regvdiBg a 
report that NASA was mvoKed in the deshuctioB of metiriah In that prior mstanoe, 
your own Geoeral Coeinsd destroyed video records of yoor appeansce bedbra tba staff of 
foe htspedor General The cvidcooeofmiscooducswu so deer that foe Chaitmas and 
Ranking Member of foe lavcetigiibotts ami Oveai|ht Subcoounstiee sent a bipetlisM 
reicRil letter to foe Depaitmeoi of Juetke sarlrirg foe proaeeonos of your Oeneral 
CounaeL 



379 


Dr. Qriffin 
Page 2 

OcL 21,2007 

We want to prevent any repeat perfoimaoce with data and records generated as part of the 
NAOMS pcx>ce». In a Septembers letter denying a press request under the Freedom of 
Information Act for the data generated through NAOMS interviews with commercial 
pilots, Associate Administrator Thomas Luedtke indicated that the data would not be 
releas^ because it is “sensitive and safety-related, [and] could m a t e ri al l y affoct the 
ptfoHc confidence in, and the commercial wdfore o( the air canien and general aviation 
companies whose pilots participated in the survey.” Given the inference fiom that 
response that at NASA commercial interests sppesr to trump the public’s tight to avistkm 
safety data, we are worried that the integrity of the data fiom NAOMS may be at risk. 

We expect to receive your immediate commitment that the relevant NASA contractors 
and stfocontractOR will be given clear, unequivocal guidance not to purge thefi records. 
Further, we expect your commitment that die records in NASA's possession will rtot be 
destroyed or otherwise compromised. 

The Committee intends to bold a hearing on this matter at the earliest possible date. 
Tberefine we ask that you accelerate the production of materials requested in the letter 
sent on October 19. Please deliver records related to any briefings or presentations given 
by Ames Research Center researchers and an answer to the question of udiy funding for 
NAOMS was cut (both of these elements arc described more fiiUy in the October 19 
letter) no later than 5 pjn. Monday, October 29, 2007. 

Further, we ask that you provide all records related to the guidance to your prime 
contractor, Battelle Memorial Institute, that it archive reooids, return them to NASA and 
then purge their own holdings on NAOMS. Please provide these materials no later than 5 
pm. Monday, October 29, 2007. 

NASA has made repeated representations, to Committee stiff in interviews as well as in 
the FOIA denial letter signed by Mr. Luedtke, that the material interests of the 
commercaal aiiline industry may be harmed by release of data developed under NAOMS. 
Please provide to the Committee any records in the possession of the agency fiom the 
commercial airline industry (carriers or representative organizations) in which foe 
concern that NAOMS data may affect their commercia] interests was communicated to 
NASA. Please provide those records to foe Committee no later than 5 p jn. Tuesday, 
November 6, 2007. 

Finally, we ask that you make a copy of all NAOMS data resulting fiom foe pilots survey 
and in foe possession of either NASA or Batellee and deliver it to foe Committee in an 
electronic fbnnsL As we wish to insure that an unadulterated record of that data be 
retained, we request foe raw data files that the Tesearchers at Ames ate supposed to be 
woiking fiom to produce their analysis of foe pilot survey. Please provide foose records 
to foe Committee no later than 5 p jn. Tuesday, Noveinb« 6 , 2007. 
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All of the requested metenels should be delivered to the offices of the Comatinee in B~ 
374 Rsyboni House Office Budding. Please provide two copies (oow for the mqocUy sod 
one for the minority). If yonr staff her any fottber qoestioiis or need additional 
infotmatioQ, please oontacl Dan Pesraoo, Investigstioiis and Oversi^ Subcommittee 
staff diiector, at (202) 225-4494, or Edidi HoUemm, Investigalive Counsel, at (202) 223- 
84S9. 


SiocaRiy, 

Wo-t_ 

BART GORDON BRAD,vniXER MARKUDALL 

fTheiiman Chsinnao Chaismaa 

Subco mm ittee oo Subcommittee on Space A 

Itrveatigationt A Ovenijlit Aenmaotics 


Rep. Ralph KaU 
Ranking Member 

Rep. F. James Senseobreaner 
Rankiiig Member 

Subcommittee on InvestigattonsA Oversigtit 


Rep. Tom Feeney 
Rmldng Member 

Subcommittee oo Space & Aeroossitica 
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ATTACHMENT 

1. The tens ‘Yeconis” ii lo be cxiostnied in the broadest aeoM and dtall mean any 
written or graphic material, however produced or rcfitodnced, of any land or 
deaciiptiao, consiating of the original and any non-identical copy {whether 
different fiom the original because of notes made on or attached to such copy or 
otherwise) and drafts and both aides thereof whether printed or recorded 
electronically or magnetically or atoned in any type of data bank, including, but 
not limited to, die following; cotrespondence, memoranda, records, summaries of 
personal conversationt or interviews, minutes or records of meetingt or 
oonfcrences, opinions or reports of oonsultania, projections, statistical statements, 
drafts, contracts, agreements, purdiase orders, invoices, confirmations, telegnqihs, 
telexes, agendas, bocdcs, notes, pamphlets, periodicals, reports, studies, 
evaluations, opinions, logs, diarica, de^ calendars, appointment books, tape 
recordings, video recordings, e-mails, voice mails, computer tapes, or other 
computer stored matter, magnetic tapes, microfilin, microfiche, punch cards, all 
other records kqX by dectronic, pbotogrsfihic, or meditnicsl means, charts, 
photographs, notebooks, drawings, plans, inter-ofBce communicstions, intra- 
office and intta-depattmental communications, transcripts, checks and cancded 
checks, bank statements, ledgets, books, records or statemeots of iccounts, and 
prices and tlimg« nmilar to any of the foregoing, however denominated. 

2. The terms ‘'idating,” ‘Tdale,” or ’Regarding” as to any given subject means 
tnytfaing that cons ti t ut es, contains, embodies, identifies, deals with, or is in any 
manner whatsoever pertinent to that sulqect, including but not limited to records 
eonoeniing the p rq wa t i on of other recoil. 
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Nstonal Aafonauoc* and 
Space Admln H li a t l oo 
Office of tfw Adiirinlatrator 
Wasnmglan. DC 20546-0001 


Oclob«29.2007 



The Hotwiable Bart Gordon 
Chaitman 

Cooiminee on Scteoce and Technology 
U.S. House of Repnsenlattves 
Washington. DC 2051S 


Dear Mr. Chairman: 

This is in fintber response to the letter of October 19, 2007, from ChaitmaD 
Miller, and your Icrter of October 22. 2007, signed jointly with Chiinnan UdaUand 
Chairman Miller, regardmg the Naiicioal Aviation Operations Monitoiing Service 
(NAOMS) and requesting several items be provided to the Cotnmitiee. We have 
previously provid^ Chairman Miller copies of the questionnaires used for pilots at part 
ofthe NAOMS survey, as requested in tte first letter. Enclosed herewith is the 
remainder of the material requested in both letters. 

As requested in the letter of October 19, enclosed is a CD with copies of 
“briefings or presentation materials that the staff at Ames gave to the Airline Pilots 
Association or other consthuem memben of the Commercial Aviation Safrny Team 
(CAST) since January 1 , 2005.” (Copies also previously given to Investigations and 
Ovetsi^ Subcommittee staff). 

The letter of OcUtber 22 requested several additional pieces of informaiitm; 

• “All records related to the guidance to your prime contractor, Battelle Memorial 
Institute, that it archive records, return them to NASA and then purge their own 
holdings on NAOMS.** EsKlosedare: 

> Docuntents relevant to the coninct provisions that deal with records 
archival and letention; 

* contract signature page (Enclosure 1); 

■ sections H. 5, R 6 and H. 10 of the contract (Eoclosme 2); 

■ section LI, which inoorpoialcs the Federal Acquisition Regulations 
clause regarding rights in data (Enclosure 3); 

■ contract task order 2, modification 4 (Enclosure 4); and 

• contract task Older 11 (Enclosures): 

> A copy of the memorandum from Battelle to its svbcontractms regarding 
the records retention policy (Enclosure 6); 
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> A copy of instiuctioiis from the NASA Geoeral Counsel regarding 
presCTvation of data (Enclosure 7); 

> A copy of an e-mail firom the Battelle AviaticD Safety Monitoring and M 
Modeling Pro g r am Manager afBrtning that NASA bat never directed 
Battelle to dettrxty the master copies of KAOMS survey results (Enclosure 
8 ). 

• Any records “ftom the commercial airiine industry (carriers or representative 
organizations) in which the concern that NAOMS data may affect their commercial 
interests was communicated to NASA.” 

> We have not found any documents te^xtnsive to this request 

• A "copy of all NAOMS data resulting fiom the pilots survey and in the possession of 
eitber NASA or Battelle.. .in electronic format” 

> Enclosed ate four CDs: Air Carrier Data April 2001 through December 
2004; Air Carrier Data Joint ImplementatioD Measurement Data Analysis 
Team (JIMDAT) Section C Supplement, ^vil 2001 through Decembw 
2004; General Aviation Data, August 2002 through April 2003; and 
NAOMS Raw Data Ait Carrier including Field Trial. 

This data is in the process of being reviewed by NASA. NASA believes (hat the 
data contains both confidential commercisl data and infonnation that could compromise 
anonymity that sh^d be redacted prior to public release. 

The raw data file is defined as the original capure of survey participants’ 
responses to a series of questions without any evaluation of the validity of the data points 
to determine if there are input errors, duplicate responses, or if the participants’ responses 
are outside of operationally possible lev^, etc. These types of errors are consider^ as 
outliers and are typically omitted fiom final data analyses after comprehensive statistical 
evaluation uoA assessmenL As such, if any data analysis is done without considering this 
routine Kientific step, the results could potentially be over or under representations of 
actual valid data. Therefore, the proces^ data have these "outliers” removed have 
also been provided for air carrier, general aviafion and JIMDAT data sets. 

To ensure that no destruction of data, iiKluding that held by sub-omitractots, 
occurred, NASA has since notified the NAOMS project management team and Battelle to 
retain all records related to the NAOMS project. Battelle has provided the same direction 
to its subcontractors. 

The letter of October 19 also requested "a written explanation of the budget 
decision to terminate support for the NAOMS project” It hu been widely reported that 
NAOMS fimding was cut or prematurely shut down. That is not the case. When the 
project originated in 1998, it was intended to contmue until 2004, as indicated in project 
briefings that were provided to various Government arxl industry audiences when the 
project began. (As mentioned above, copies of these briefings are enclosed. Later 
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brieSnp indicatad an exteosioa to 2O0S.) Funding was extended Omnigh 2006 to allow 
fix ttansitian of the methodology and final documentation. 

However, the overarching goal of trying to develop methodologies that enable 
data-driven system safety analyses is one dtat NASA continues to embrace in its current 
Aviation Safety Program, in close partnership with the FAA, industry, and aitidrmia In 
order to oontinuaUy and significantly reduce the accident rate to meet the expected 
growth of the Next Generation Air Transportatioo System (NextOen), it is imperative to 
develop a robust safety information system that discovers safety precursors b^oce 
accidents occur. Accomplishing (his will require the ability to combine and analyze vast 
amounts of data from many varied sourees to detect and act on new safety threats. 

NASA and the FAA are combining their unique skills and resources under clearfy 
defined roles and reqraosibilities to address this challenge. In order to ensure that the 
technology is efifoctively (ransitiooed between organizatioos, a prog r a m plan has been 
developed and is being executed. The initial response to (his approach fiom the 
stakeholder community has been vety positive. 

I believe this material it fully responave to your requests. 1 would be happy to 
diacass this matter fimber as desired at your convenienoe. 



Administrator 


Enclosures 


oc: 


The Honorable Ralph Hall 
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NNACMOeOTTOR SECTION H 

H.S. MANAGEMarr ANO PROTECTK>N Of DATA (ARC 52.227-63) (JUL 1988) 

(4 In the performance of IMS oontract II is a n HOi pa ted that the Comractor msy have 
access le, be furnished, use. or generate the foBowIng types of data (recorded 
Infamuilan): 

(1 ) data submitted to the Govemmont with hiBted rtgtits or restitcted rights notloes: 

(2) dais of third parties which the Govamment has agreed to handle under protective 
arrangements: and 


(3) data generated by or on behalf of Bie Govommsnl. vrhich the Oovemmmt Inlands to 
control lha use and dlseemtoaiton thereof. 

(b) In order to provide manegement appropriate for protecting the inlerasts of the 
Government and other owners of sudi deaths Contractor agrees with respect to date 
In category (aXt) above, and wflh respect to any data in categortes (aX2) «td (aX3) 
when so ktondflad by the Coniractkig Officer, to: 

(1) use and disdoee such data only to the extent necessanr to perform the work 
repdred under this oontract, with parttoilar emphesis on restrtotsig (tsdosurs of the 
dais to those persons who have e dellniM rtead for the data In ordar to perform under 
thtooonliBCi; 

(2) nd reproduce Bie data unlass raproduoficnof Ihedata Isapedficaly permMed 
els aw here In Ote cenlra ct or by tbe Contracdng Officer 

(3) refrain from disdoeing the data to Brird parBas without the wrffien consent of Bie 
Contracting Officer, and 

(4) return or deflver the data indudlng all oopias thereof to the Contracting onioer or hit 
designated recipient when requeeted by the Contracdng Olllcar. 

(END OF CLAUSE) 

H.e. HANDLING OF DATA (ARC 52227-66) (JUN 1886) 

(a) Paragraph (dXI) of the *Rights In Deta-OeneraT clause of this oontract permits Bie 
Gover nm ent to resiM the Contractor's rIgM to use. release to others, reproduce, 
dIsMbute, or publish any t]3la first produced or specifically used by Bie Oonlractor in Bie 
performance of the oonlraei provMed such restrtcdon is aigwessfy set forth in the 
contracL Pmuant to BBi authority, the foocwing iBstrtcQons Shan apply to such data 
and Bhal be Included, in substance. In sV subcontracts: 

(b) Data spectficalty used. 

(1) In the perfotmanoe of this contracL R Is anttcipated the Contractor may have access, 
or be furnished, dafo (Ihctuding finanoisl, edmimstrallve. cost or pricing, or managament 
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tnformafion as well as ledinical data or com^Mitar sollwara) of IMrd parties wtiich the 
Government has agreed (o handle under protective arrartgements. as well as such 
Government data for wtiich the Government intends to control the use and 
dissemination. 

(2) In order to protect the interests of the Government and the owners of such data. Ihe 
Corrtrador agrees, with respect to such IhN patty or Government data that ia either 
marked with a restriclive legetrd or specifically Idertlllled In tMs contraci or in writing by 
the Contracting Officer as being subject to this ctause, to use and disclose such data 
only to the extent necessary to perform the work required under this contract, pradude 
disdlasure of such data outside the Contrador's otganizaSon, and return or dispose of 
stxh data as directed by the Contradlrtg Officer when the data Is no longer needed for 
oonbad performanoe. 

(3) Notwlthstandng (2) above, Ihe Contractor (hall not be restrictad In the use and 
disclosure of any data that becomes generaSy avaiiabie without breach of this clause by 
tNs Contractor, Is known to or is developed 1^ the Contractor IndependenOy of any 
dtsdosure of proprietary, restolctad, or oonfid^al data hereunder, or is rightfully 
reodved by the Contrador from a third party wIBiout restriction. 

(c) Data Hcst produced. 

Data (irsi produced by the Contractor ur«der this ooniract may indude data for which Ihe 
Government wants to control the use and diseemhiation. Tlie Contracting Officer may 
require, or INs conirad may presanfly specify, that the Contractor apply nsstricbve 
legetxis to such identIfiBd data prior to deSve^ to the Government, or to third parties at 
the Govenanenl's direcOon, that resirtot Ihe use and disclosure of the data by any third 
party recipient However, sudi restrictive legends shall In no way affect the Contractor's 
or the Govemtrrenfs tights to such data as provided In ihe ’Righto hi Data-OeneraT 
clause of this conbact 


(END OF CLALISE) 


H.7. SEVERANCE PAY (ARC 52,231-90) (MAY 1993) 

In oonjunctton with FAR 31,20S-6<g), Ihe severance pay cost shall not exceed 40 hours 
pay for each year of employment per employee up lo a maximum of 80 hours per 
elgible employee. Severance cost eligIbiKy oomputafion for reimbursament shall also 
be imited to only the period of employment on the service contract at Ames Reseamh 
Center. In no event shall Ihe Govammertt relmburae Ihe Contractor tor severance cost 
tor employees who voluntarily accept emptoyment in piaca with Ihe succeeding 
conira^r within ninety (90) days aftsr comptelton of the current contract 

(END OF CLAUSE) 
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H.8. SUBCONTRACTING. DATA NOT FIRST PROOUCEO UNDER THE 

CONTRACT AND REPRESENTATION OF LIMITED RIGHTS DATA AND 
RESTRICTED RIGHTS SOFTWARE (ARC 52-227-97) (OCT 2004) 

It Is strongly rscontmanded that ttis Contractor flow down ffw data rtgtits prov tii on s of 
this oontract to krwsr Mr subcontractors to ensure that It can ftdfH Its data rtgiits 
obli gati ons to the GovemmenL See Clause FAR 52.227'14{h). Rights h D»t»— 
General. The Contractor shsi be held responsible to obtain rights Ibr the Government 
where M fsHs to fulfil such obigalions. 

Offerors are remirtded that as required by Clause FAR S2.227-1 4(c)(2). the Contractor 
must obtain Osttraciing Oflioer approval before incoeporating any data not Rrst 
produced under the Contract imo data deivsred under ihe contracL Before dsavsrlng 
such data, the Contractor must identify It and grant the Government or acquire on Its 
behalf, the broad licenses required by subparagraph (c) at the Rights In Dtts — General 
clause. 

The Contractor shall make the representation required by FAR S2.227-1S lor each 
contract task order. On a case-by-case basis, the Government will insert the purpoces. 
rights or limitations under which Ihe Government can use Limited Rights Data and 
Restricted Rights Software Into the alternate clauses II and III of FAR 52.227-14. 

(END OF CUUSE) 


H.9. INFORMATION INCIDENTAL TO CONTRACT ADMINISTRATION 
(ARC 52.227-98) (OCT 2004) 

NASA Shall have untmitad rights In kiformabon Incidental to contract sdmlnistrBtlon 
tndudng adminlstTative and management Inionnetion created by the CotTtractor and 
specdlad lor delivery to NASA in performance of Ihe oontracL expressly excluding 
financial Infarmabon. SpecMcaRy. NASA shsI have the right to neleasa such 
edministiatlve and management Informalian to any third party to sabsfy NASA's 
requirements. 


(ENOOF CLAUSE) 

H.10 DATA RIGHTS— HANDUNG OF DATAIMANAGEMENT 4 PROTECTION OF 
DATA 4 SPECIAL WORKS 

The Contractor Is hereby Insbucted that the catagortes of data idantlfled below are 
subject to the non-disclosure, handllrtg and other requhed obTigatlons of ARC 52.227- 
93 (Management and Protection of Dala)(Clause H.5) and ARC S2.227-96 (Handling of 
DataKCtause H.6) of the contracL 


Please revlsw the lequiremenls of these dauses which indude the following obligations: 

30 
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(1) use and disclose such data only to the extent necessary to pefform (he worir 
required under this contract, wtth particular emphasis on restricting disdosure of the 
data to those persons who have a definite r>eed for the data in order to perform under 
this contract; 

(2) the Contractor agrees, with respect to such third party or Government data that is 
either marked wtth a restrictive legWxt or spectficaify identified In this contract or In 
witting by the Contractirrg Officer as betrrg subject to this clause, to use and disclose 
such data only to the extent necessary to perf^ the work required under (his contract, 
preclude disclosure outside the Contractor's organization, and return of such data as 
directed by the Contracting Officer when the data is no longer needed for contract 
performance. 

Categodes of data Idanttfled under this coniract; 

Any flight recorded data from FOQA programs 

Any radar data from PDARS programs 

Any safely report data from Aviation Safety AcSon Programs 


The Contractor is hereby directed to assert copyright, or authorize assertion thereof. In 
special works data and to assign, or obtain the assignmerTt of. such copyright to the 
Government or Its designated assignee In accordance wKh Clause 52,2^-17 Rights in 
Data-Spedal Works, The dbecfion appTies to software extensions of Morning Report to 
air traffic oontrot data and distributed national FCQA archive software. 

(END OF CLAUSE) 

(END OF SECTION) 
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PART II • CONTRACT CLAUSES 


SECTION I - CONTRACT CLAUSES 


1.1 . LISTING OF CLAUSES INCORPORATED BY REFERENCE 

NOTICE; ThN contract Inoorporatos one or more clauses by refersnoe. witb the same 
force sfxl effect as V they were gNen ki foil text Upon rsquM, the Contracting Oflfosr 
win make their fol text avaiaMe. Alao, the (ul text of a clause may be aoceased 
s l ecao nlc B ff y at Ms/these aOdressfes); 



1. FEDERAL ACQUISmON REGULATION (48 CFR CHAPTER 1) 

CLAUSE 

NUMBER DATE TITLE 

52J02-1 JUL 2004 DEFWITIONS 

S2.20M APR 1984 GRATUtTlES 

52.20M APR 1984 COVENANT AGAINST CONTINGENT FEES 
52 J03^ JUL 1995 RESTRICTIONS ON SUBCONTRACTOR 

SALES TO THE GOVERNMENT 

52.203- 7 JUL 1995 ANTHOCKBACK PROCEDURES 

52.203- 8 JAN 1907 CANCELLATION. RESCISSION AND 

RECOVERY OF FUNDS FOR ILLEGAL OR 
IMPROPER ACTIVITY 

52.203- 10 JAN 1997 PRICE OR FEE ADJUSTMENT FOR lUEGAL 

OR IMPROPER ACTIVITY 

52.203- 12 JUN2003 UMITATION ON PAYMENTS TO INFLUENCE 

CERTAIN FEDERAL TRANSACTIONS 

52.204- 4 AUG 2000 PRINTED OR COPIED DOUBLE-SIDED ON 

RECYCLED PAPER 

52.204- 7 OCT 2003 CENTRAL CONTRACTOR REGISTRATION 
52.209-8 JUL 1 995 PROTECTING THE GOVERNMENTS 

INTEREST WHEN SUBCONTRACTING WITH 
CONTRACTORS DEBARRED. SUSPENDED. 
OR PROPOSH3 FOR DEBARMENT 

52.211- 5 AUG 2000 MATERIAL REQUIREMENTS 

52.211- 15 SEP 1990 DEFENSE PRIORITY AND ALLOCATION 

kaya 12 


gjucubiijfi^ 3 
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52.215- 2 

52.21 5- 6 
\ 

JUN 1999 
OCT 1997 

52.215-10 

OCT 1997 

52.215-11 

OCT 1997 

52.215- 12 

52.215- 13 

OCT 1997 
OCT 1997 

52.215-15 

JAN 2004 

52.215-18 

OCT 1997 

52.215- 19 

52.216- 7 

OCT 1997 
DEC 2002 

52.216- 8 

52.217- 8 
52.219-8 

MAR 1997 
NOV 1999 
OCT 2000 

52.219-9 

JAN 2002 

52.219-16 

JAN 1999 

52.222-1 

FEB 1997 

52.222-2 

JUL1990 

52.222- 3 

52.222- 21 

52.222- 26 

52.222- 35 

JUN 2003 
FEB 1999 
APR 2002 
DEC 2001 

52.222-36 

JUN 1998 

52.222-37 

DEC 2001 

52.223- 6 

52.223- 14 
52.225-1 
52225-13 

MAY 2001 
AUG 2003 
JUN 2003 
DEC 2003 


REQUIREMENTS 

AUDIT AND RECORDS - NEGOTIATION 
ORDER OF PRECEDENCE - UNIFORM 
CONTRACT FORMAT 

PRICE REDUCTION FOR DEFECTIVE COST 
OR PRICING DATA 

PRICE REDUCTION FOR DEFECTIVE COST 
OR PRICING DATA - MODIFICATIONS 
SUBCONTRACTOR COST OR PRICING DATA 
SUBCONTRACTOR COST OR PRICING 
DATA- MODIFICATIONS 
PENSION ADJUSTMENTS AND ASSET 
REVERSIONS 

REVERSION OR ADJUSTMENT OF PLANS 
FOR POSTRETIREMENT BENEFITS (PRB) 
OTHER THAN PENSIONS 
NOTIFICATION OF OV/NERSHIP CHANGES 
AUOWABLE COST AND PAYMENT Insert *30 
days' In Paragraph (sKS) 

FIXED FEE 

OPTION TO EXTEND SERVICES 
UTIUZATION OF SMALL BUSINESS 
CONCERNS 

SMAa BUSINESS SUBCONTRACTING PLAN 
(ALT lIXOCT 2001) 

LIQUIDATED DAMAGES - 

SUBCONTRACTING PLAN 

NOTICE TO THE GOVERNMENT OF LABOR 

DISPUTES 

PAYMENT FOR OVERTIME PREMIUMS (INSERT: 
•scr IN PARAGRAPH (a)) 

CONVICT LABOR 

PROHIBITION OF SEGREGATED FACILITIES 
EQUAL OPPORTUNITY 
EQUAL OPPORTUNITIES FOR SPECIAL 
DISABLED VETERANS, VETERANS OF THE 
VIETNAM ERA AND OTHER SPECIAL 
VETERANS 

AFFIRMATIVE ACTION FOR WORKERS WITH 
DISABILITIES 

EMPLOYMENT REPORTS ON SPECIAL 
DISABLED VETERANS. VETERANS OF THE 
VIETNAM ERA, AND OTHER ELIGIBLE 
VETERANS 

DRUG-FREE WORKPLACE 
TOXIC CHEMICAL RELEASE REPORTING 
BUY AMERICAN ACT-SUPPUES 
RESTRICTIONS ON CERTAIN FOREIGN 
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52.227- 1 

52.227- 2 

52.227- 14 


-52.227-16 

52.227-17 


52.227- 19 

52.227- 23 

52.228- 7 

52.230- 2 

52.230- 3 

52.230- 6 

52.232- 9 

52.232- 17 

52.232- 18 

52.232- 20 

52.232- 22 

52.232- 23 

52.232- 25 

52.232- 34 


52.233- 1 

52.233- 3 

52.237-3 

52.239-1 

52.242- 1 

52.242- 3 

52.242- 4 

52.242- 13 

52.243- 2 


SECTION I 


JUL1995 
AUG 1996 
JUN 1987 


JUN 1987 
JUN 1987 


JUN 1987 

JUN 1987 

MAR 1996 
APR 1998 
APR 1998 

NOV 1999 

APR 1984 

JUN 1996 
APR 1984 
APR 1984 
APR 1984 
JAN 1986 
FEB 2002 

MAY 1999 


JUL2002 
AUG 1996 

JAN 1991 
AUG 1996 
APR1984 
MAY 2001 
JAN 1997 
JUL1995 
AUG 1987 


D1 fPr'UAQPQ 

AUTHORIZATION AND CONSENT 
(ALTERNATE I) (APR 1984) 

NOTICE AND ASSISTANCE REGARDING 
PATENT AND COPYRIGHT INFRINGEMENT 
RIGHT IN DATA - GENERAL (ALT II) (JUN 
1987) (ALT Ul) (JUN 1987)(AS MODIFIED BY 
NFS 1852.227-14, RIGHTS IN DATA- 
GENERAL) 

ADDITIONAL DATA REQUIREMENTS 
RIGHTS IN DATA - SPECIAL WORKS (wBi 
subparagraph (e) Indemnity deleted from this 
clause) 

COMMERCIAL COMPUTER SOFTWARE - 
RESTRICTED RIGHTS 

RIGHTS TO PROPOSAL DATA (TECHNICAL) 

, (INSERT: PAGES , DATED ) 

INSURANCE-LIABILITY TO THIRD PERSONS 
COST ACCOUNTING STANDARDS 
DISCLOSURE AND CONSISTENCY OF COST 
ACCOUNTING PRACTICES 
ADMINISTRATION OF COST ACCOUNTING 
STANDARDS 

UMITATION ON WITHHOLDING OF 

PAYMENTS 

INTEREST 

AVAILABILITY OF FUNDS 
UMITATION OF COST 
UMITATION OF FUNDS 
ASSIGNMENT OF CLAIMS 
PROMPT PAYMENT (ALTERNATE I) 

(FEB 2002) 

PAYMENT BY ELECTRONIC FUNDS 
TRANSFER- OTHER THAN CENTRAL 
CONTRACTOR REGISTRATION (Insert "No 
later than 15 days prior to submission of the first 
request for paymertt* in paragraph (bK1)) 
DISPUTES (ALTERNATE I) (DEC 1991) 
PROTEST AFTER AWARD (ALTERNATE I) 
(JUN 1985) 

CONTINUITY OF SERVICES 
PRIVACY OR SECURITY SAFEGUARDS 
NOTICE OF INTENT TO DISALLOW COSTS 
PENALTIES FOR UNALLOWABLE COSTS 
CERTIFICATION OF FINAL INDIRECT COSTS 
BANKRUPTCY 

CHANGES-COST-REIMBURSEMENT (ALT 
IVXAPR 1984) 
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52^44-2 AUG 1998 SUBCONTRACTS (ALT IXAUG 1998) 

52^44-5 DEC 1996 COMPETITION IN SUBCONTRACTING 

52^45-1 APR 1984 PROPERTY RECORDS 

52.245- 5 MAY 2003 GOVERNMENT PROPERTY (COST- 

REIMBURSEMENT, TIME-AND- MATERIAL. 
OR LABOR-HOUR CONTRACTS) 

52.245- 19 APR 1984 GOVERNMENT PROPERTY FURNISHED "AS 

IS- 

52.247-1 APR 1984 COMMERCIAL BILL OF LADING NOTATIONS 
52.2486 SEP 1996 TERMINATION (COST-REIMBURSEMENT) 

52.24814 APR 1984 EXCUSABLE DELAYS 

52.251-1 APR 1984 GOVERNMENT SUPPLY SOURCES 

52.2581 JAN 1991 COMPUTER GENERATED FORM/S 


II. NASA FAR SUPPLEMENT (48 CFR CHAPTER 18) CLAUSES 
CLAUSE 

NUMBER DATE TITLE 


1852.20870 

1852.204-76 


1852.21875 
1852.21874 
1852^1875 

1852.21876 

1852.227- 14 

1852.227- 17 
1852.23870 
1852.24873 


JUN 2001 DISPLAY QF INSPECTOR GENERAL 
HOTUNE POSTERS 

JUL 2002 SECURITY REQUIREMENTS FOR 
UNCLASSIREO INFORMATION 
TECHNOLOGY RESOURCES (INSERT: *30 
DAYS- IN PARAGRAPH (c)) 

DEC 1988 PAYMENT OF FIXED FEE 
SEP 1990 USE OF RURAL AREA SMAa BUSINESSES 
MAY 1999 SMALL BUSINESS SUBCONTRACTING 
REPORTING 

JUL 1997 NASA 8 PERCENT GOAL 
AUG 1997 RIGHTS IN DATA- GENERAL 
AUG 1997 RIGHTS IN DATA- SPECIAL WORKS 
FEB 2003 CENTER FOR AEROSPACE INFORMATION 
OCT 2003 FINANCIAL REPORTING OF NASA 
PROPERTY IN THE CUSTODY OF 
CONTRACTORS (Insert NASA Ames Researcti 
Center. M/S 2582, Moffett Beld. CA 94038 
1000 ) 


(END OF CLAUSE) 


1.2. OMBUDSMAN (NFS 1852.215-84) (OCT 2003) (ALTERNATE I) 

(JUN 2000) 

(a) An ombudsman has been ^polntad to hear and facilitate the resolution of concerns 
(nim offerors, potential offerors, and contractors during the preaward and postaward 
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Batreiie 

n> Burtnrca a( lanvavtioai 

KUOMiMM 
Olv 43301 ‘I**] 

SopMoibor 10, 2007 I4i4n4.4434 h.i<)4i<9uiu 

Botullf Propotot No. OP469j9 


Mt. Melitialynn Pericinc 
Contnebri OfBcot 
MS 241-1 
NASA ARC 

MofTea FmIO. CA 94025-IOOO 


Dear Ma Petkini: 


Barielte Memorial Inaulvie it pleased lo lubroti tliii propoul lo support Coocner Number 
NNA0SAC07C. CTO *I 1, etuiUed 'ASMM Phsie Ool* under (be NASA ASMM proram 

This proposal is tubmilied on a con phis fixed fee basis Jbr a loral estimated cost orSl9,t32, 
which includes a fixed fee of SI,I3& Bsitellewill invoice inettrted cotta on a monthly basis 
This proposal is valid A» 30 ds]^ Acceptiace after Out date will be by agreement with Banelle. 

please direct questions ofa business or contcactual nstiae lo Mr. William & Jones at (614) 424- 
70t9. Techni^ questions ahoold be direcied ID Mr Loren Rosenthal at (6S0D 960-6010. 


Sinecrely, 


Willtinn £. Jones L-/ 

CootmetiDg Officer 




WEiup 

findocure 


n« aesQM «r oMMon negeM ew naie** 
er ^ aaa iv a^r n«POM oAar 
•ov M a faaji «f. «r n aiwinaaaa r« 
ewinaba— NBKiiea e aifTaft m aae 
iM MMitfaia / 1 • aftiai— a aa» afi<a 
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Subject Reaenlion of NAOMS Records 
Dbs: Hiu. IS Oot 2007 OWI :04 -0700 
Thread Topic: Relcasioo ofNAOMS Records 
Priority; Utgeol 

Ffsni: 'Lorca Rasealha]* <>gi»nJoiendiil®)Mnrtlemvca.or|> 

To: *Haber. Dan^' <HabcrD@BATrELLE.ORO>. <IL>beriSOodd@coiacaiu>el>. 
*Joo Kcondck* 4roenielu@aunford.edii>, <iiiailvtt^iiacapeadeaoesxoin>, 
'Petrynua. Usamas A* ' 4 »a.ienyinan@pnl.gov>, 

'AUea Carter* <aeanct @giaiL s»tjasas<^. 

'loon Cssi* <cwtjs@BATTELLE.ORO>, 

■piaceU, JadflSa M’ <|iiiiceUJ@BATTELLEORC> 

Cc: ‘Olsoa. Katby- <alsoiifc@BATTEU.£.ORa>, 
*isutI^<lndag.C.Stitlcc^iaaa.|ov>, 

■Dave WHliaras’ <srilliaiiid^ATTELLE.ORO> 


To all: 

BaneUe is in recdpl of a letter Snm die US Hoose of Rcpreaeiitilrvea Cooaiiittee on 
Sdeaoe and Techaology pestaisiQg so dse NAOMS ptqect I Iieve already 
oaomiuiiscatcd to yea the need to aastae yoo do not delete any protect related data. By 
dna e-nail I am ato fiirtnrditig the rcquiiciiiait we have reoei^ in die letter. Pleoee 
aiaiiie you fiilly cooply with the requireineiita set farsb below. 

Tbatika for your oocpcritioa. 

Loren 

*By diis letter, we ace directiiig BatleUe Inadane, ha cmploycca and subcooliacaon to 
retain all master copies, and iccotidary copies in die pciairation of Battelle or any of its 
e inp i o ye ca or suhc e otraclora. Fialber, we arc directint Baciellc Insticute, Ha etnployeea 
and subcoatraotota to retaan all records (as defined in die A tiadimu it) relating to the 
NAOMS project and the survey of airline pilots conducted under contract witfa NASA. 
Dcatraction of documonts requested aa part of a Congiesaioiial inqniiy is a vMadon of 
criminnl federal law (It U.S.C I SOS), and iheaedacinneota were requested yesterday in 
a lence to NASA from Oiainnm Ooidon. Miller and UdaU. (Leder dated OA 22,2007, 
finsn Cbadrmen Goidcn, Miller and UdaU to NASA Adininiaijsict Micbael OtiflliL) 

If any reootda heve already been dcattoyed, please provide a list of tbe docnmenla 
deaaoyed and the dale of destructico.* 

ATTACHMENT 

I. Tile term ‘itcotda* ia 10 be construed in the broadest sense and shall mean any 
written or gtaphic material, however produced or reproduced, of any Idad or descriptioii, 
cenaiating of the original and any noo-ideatkal copy (wbeilier ditfe e itt fiuin the original 
because of notes made on or anariied to such copy or otherwise) and drafts and both sides 


Lf 
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thoeof^ w4iether printed or reooided electrooically or magnetically or stored in any type 
of data bank, including, but not limited to, the following; conespondence, memoraoda, 
reconU, summaries of .oenonal conversations or interviews, minutes or records of 
meetings or conferences, opinions or reports of consultant, projections, statistica] 
statements, drafts, contracts, agreements, purchase orders, invoices, confirmations, 
telegraphs, telexes, agendas, books, notes, pamphlets, periodicals, reports, studies, 
evaluations, opinions, lo^ disries, desk c^endats, qipointmern hordes, tape recordings, 
video recordings, e-mails, voice mails, computer tapes, or other computer stored matter, 
magnetic tapes, microfilm, microfiche, pun^ cards, ail other records kept by electronic, 
photographic, or mechanical means, chans, photographs, notebooks, drawings, plans, 
inter-ofiBcc communicafions, intra-oCSce and intra-dqurtmoital communicationt, 
transcripts, checks and canceled checks, bank statements, ledgers, books, records or 
statements of accounts, and papers and thiitgs similar to any of the foregoing, however 
denominated. 

2. The terms "relating,* 'relate,* or "regarding* as to any given subject means 
anything drat constitutes, contains, embodies, identifies, deals with, or is in any manner 
whatsoever pertioent to that subject, including but nor limited to records concerning the 
preparation of other records. 


KatiiyA. Olson 
Assistant General Counsel 
614-424-6580 
614-458-6580 (fax) 
olsonk@baaelie.org 


CQjqaBEyTlALITY ^9T^CT 

This information is fiom the Law Department of Battelle Memorial Institute. This 
mess^ is intended only for the use of the individual or entity to which it is addressed, 
and which may contain informatioa that is privU^ed, confidential and/or otherwise 
exempt fiom disclosure under applicable W. Iftfae reader ofthis message is not the 
intended recipient or the employee at agent responsible for delivering the message to the 
intended recipient, any disclosure, diss^nation, distribufion, copying or other use of 
this communication or its substance is prohibited. Ifyou have received this 
communication in error, please renirn to the sender and delete from your computer 
system. THANK YOU. 




403 


Nwbi 

Sant: 

Tat 

Ce 


OODbir 21, 2007 U:S3 AM 

Samdt TtaMi W. (MK-OU Ttanvn^t^ Umtn K. (NQ^COOO) 

PMccn. ft Mw {H^««W0X; SMM llr <MQ4«00e): S«<tP^ Ki» TTionii* (HQ4MB0U SpMr. 
K^iWi ■ (MQ4M Bg0j 

CongmaonM HiMn^ • Pntta Into 


Tom/Sumam 


In ■ liOsr daM 22 Oct tia CongreM Kss sMatf (hat Me «A ba a naanno on ow NAOMS 
tMue erd hat dlreeied that al reitvant Into te satoguarded 



IdlZ^OordM. 

MMr.uaia.. 


I need yoir idervanaon to artara thto (Ms mastage to pcasacva gate out to al nacaaaary psftlat 
Tonv ragardtoas of what the contr a ct af«i Battelft cailt tor twm to do. pleas a have Via COTR gal 
In touch wlh ffwm ASAP and raouaat Viat they both prasarva avarything and that they gat In 
touch wVh vkhatawar auba they had and tol tham to hold on to al data itotf furtttor dradad 

Sumara: can you go through Via procuramant channala hart and aaa If Viay hava a play In tots. I 
juto want to maica aura (hat wa oovar al the bases! 

Thanks. 


Ittehaal C. Vtoioflay 
MASA Oanaral Cottoial 


aOQESXraaCSW 
Wa eh ngton. DC 20S4e 
202.358.2450; FAX 2Q2.3SS.2741 
MtohaalCVPxilavttoaiaQov 

Tkli itiniiriinf. Inr hiitlig anj aTsrhmuiTi . rnnttini infhrmalira Ibaf ii rnnflrt t la l. pfranwl h) flu 
iac»a«y<tMat or other toP^kabk privilun, «r couniiMito oca-pdbbc toftnnaboa. It b iniMited oaly tor 
tot dtbfBaiad rtelptoafKii If you in dm an bsiodtd rtebibai oTtob totomu i ioo . pltait afta apprcgr bit 
mpi to dtMro y tob docoBwl hi to tatfmy id aMify toe wottor of in d ni a Lik a. Uia. d iwibakiii . 
dbatoadoa. or rigrodurtlon of toll totonaadop by intnira liil rselpkna b aM aahnrlffiil id way be 
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Subject Secure Retention of NAOMS Data 
Date; Mon. 22 Oct 2007 10;23:13 -0700 
Priootv; Urgent 

From; ■Loren Koeenthar <loren.ro8enm8l@battei)emvca.org> 

To: 'istatlof' <trving.C.StalJef@n8sa.gov>, 

*Mary Connors* <Mary.M.Connor8^asa.gov> 

Unda Conneir <Unda.J-Connell@r>asagov> 

Cc 'Dave WWams* <wi1ltamd(9battetle.org>, 

'Dennis Nelson' <nelsondb@batielie.org>, 

'Alrnt Carter' <acatter@mail.arc.nasa.gov>, 

'Kimberly Salazar* <kimberty_salazar®balt^mvca.org>, 

'Haber, DanieT <HaberD@battelie.org> 

Al- 

The purpoee of this email is to affirm, at NASA request that NASA has never 
directed Batteile to destroy the master copies of NAOMS survey results data nor 
has Batteile taken such action. Master copies of all NAOMS survey results are 
maintained by Batteile In Mountain View, CA on CDs and other backup media 
Copies of the CDs have also been conveyed to NASA Ames. 

NASA has directed Batteile to recover, or ensure the secure destruction, of any 
secondary copies of the NAOMS data that might be held at locations outside of 
Mountain View. This includes any copies held by present or past BstteMe 
NAOMS subcontractors. The purpose of this latter action is to ensure that 
NAOMS conforms to NASA data security requirements. The essential goal is to 
bring all NAOMS data to a single, secure location managed by NASA. Batteile is 
in the process of taking this action now as part of the ASMM contract phase-out 
process (NAOMS prcjeci work has been accomplished under the A^M 
contract) 

-1-oren Rosenthal 

Batteile ASMM Program Manager 





